[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:06 dist-test-slave-f9rl krb5kdc[13288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:06 dist-test-slave-f9rl krb5kdc[13288](info): set up 2 sockets
Sep 25 10:25:06 dist-test-slave-f9rl krb5kdc[13288](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:25:07.375473 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:07 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:07.554455 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:41245
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:07.890705 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:07.891184 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:07.891373 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:07.898926 13301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:07.899034 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:07.899158 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:07.899266 13301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:07.899351 13301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:07.909700 13301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:07.911352 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:07.912930 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:07.928830 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:07.929244 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:07.929531 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:07.930042 13301 server_base.cc:1034] running on GCE node
I20260925 10:25:07.930955 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:07.933087 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:07.934347 13301 hybrid_clock.cc:648] HybridClock initialized: now 1790331907934275 us; error 56 us; skew 500 ppm
I20260925 10:25:07.934876 13301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:07 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331907, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:07.938238 13301 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:07.940632 13301 webserver.cc:460] Webserver started at http://127.12.248.126:33705/ using document root <none> and password file <none>
I20260925 10:25:07.941740 13301 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:07.941897 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:07.942340 13301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:07.945979 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "c6a9790ca6874fb8bf278eb669cad775"
format_stamp: "Formatted at 2026-09-25 10:25:07 on dist-test-slave-f9rl"
server_key: "77cb2de1c8dc33488e2c2b85569021ed"
server_key_iv: "afdec20954140b1c4aea5eba170b9a55"
server_key_version: "encryptionkey@0"
I20260925 10:25:07.946834 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "c6a9790ca6874fb8bf278eb669cad775"
format_stamp: "Formatted at 2026-09-25 10:25:07 on dist-test-slave-f9rl"
server_key: "77cb2de1c8dc33488e2c2b85569021ed"
server_key_iv: "afdec20954140b1c4aea5eba170b9a55"
server_key_version: "encryptionkey@0"
I20260925 10:25:07.954139 13301 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 10:25:07.958382 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:07.959997 13301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:25:07.960340 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "c6a9790ca6874fb8bf278eb669cad775"
format_stamp: "Formatted at 2026-09-25 10:25:07 on dist-test-slave-f9rl"
server_key: "77cb2de1c8dc33488e2c2b85569021ed"
server_key_iv: "afdec20954140b1c4aea5eba170b9a55"
server_key_version: "encryptionkey@0"
I20260925 10:25:07.960570 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:07.987416 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:07.992900 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:07.993167 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:08.013506 13301 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:41245
I20260925 10:25:08.013509 13367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:41245 every 8 connection(s)
I20260925 10:25:08.015460 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:08.020153 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:08.022989 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13301
I20260925 10:25:08.023418 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:08.026180 13281 external_mini_cluster.cc:1428] Setting key 5de107cbe2f61962a40601af7cba0bc7
I20260925 10:25:08.036441 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775: Bootstrap starting.
I20260925 10:25:08.040174 13368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:08.041378 13368 log.cc:824] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:08.044888 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775: No bootstrap required, opened a new log
I20260925 10:25:08.053637 13368 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } }
I20260925 10:25:08.054118 13368 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:08.054299 13368 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a9790ca6874fb8bf278eb669cad775, State: Initialized, Role: FOLLOWER
I20260925 10:25:08.055279 13368 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } }
I20260925 10:25:08.055704 13368 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:08.055879 13368 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:08.056064 13368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:08.058403 13368 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } }
Sep 25 10:25:08 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:08.059319 13368 leader_election.cc:302] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6a9790ca6874fb8bf278eb669cad775; no voters:
I20260925 10:25:08.060155 13368 leader_election.cc:288] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:08.060525 13373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:08.061601 13373 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [term 1 LEADER]: Becoming Leader. State: Replica: c6a9790ca6874fb8bf278eb669cad775, State: Running, Role: LEADER
I20260925 10:25:08.062527 13373 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } }
I20260925 10:25:08.063599 13368 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:08.067610 13375 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a9790ca6874fb8bf278eb669cad775. Latest consensus state: current_term: 1 leader_uuid: "c6a9790ca6874fb8bf278eb669cad775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } } }
I20260925 10:25:08.067565 13374 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6a9790ca6874fb8bf278eb669cad775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a9790ca6874fb8bf278eb669cad775" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 41245 } } }
I20260925 10:25:08.068063 13375 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:08.068117 13374 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:08.074815 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:08.032497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.12.248.126:41245)
0925 10:25:08.033673 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:08.033708 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:08.035386 (+ 1678us) server_negotiation.cc:407] Connection header received
0925 10:25:08.044972 (+ 9586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:08.044993 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:08.045945 (+ 952us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:08.046509 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:08.049726 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.051127 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.052769 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.053525 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.053831 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:08.059983 (+ 6152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:08.060052 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:08.060091 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:08.060138 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:08.064494 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:08.065782 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:08.065793 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:08.065841 (+ 48us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:08.066163 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:08.066798 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:08.066805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:08.066811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:08.067762 (+ 951us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:08.068166 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:08.072032 (+ 3866us) server_negotiation.cc:299] Negotiation successful
0925 10:25:08.072605 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":473,"threads_started":1}
I20260925 10:25:08.084654 13380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:08.091223 13380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:08.103616 13380 catalog_manager.cc:1347] Generated new cluster ID: 4b506291b1e34f06ba3dc629d44f15dd
I20260925 10:25:08.103765 13380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:08.126791 13380 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:08.128297 13380 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:08.141736 13380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6a9790ca6874fb8bf278eb669cad775: Generated new TSK 0
I20260925 10:25:08.142648 13380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:08.185365 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:08.537598 13394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:08.538141 13394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:08.538435 13394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:08.550165 13394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:08.550282 13394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:08.550400 13394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:08.560779 13394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:08.562211 13394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:08.563520 13394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:08.577889 13399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:08.577863 13402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:08.577943 13400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:08.579226 13394 server_base.cc:1034] running on GCE node
I20260925 10:25:08.579847 13394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:08.580833 13394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:08.582038 13394 hybrid_clock.cc:648] HybridClock initialized: now 1790331908581987 us; error 39 us; skew 500 ppm
I20260925 10:25:08.582424 13394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:08 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:08.586017 13394 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:08.587697 13394 webserver.cc:460] Webserver started at http://127.12.248.65:46259/ using document root <none> and password file <none>
I20260925 10:25:08.588439 13394 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:08.588574 13394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:08.588912 13394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:08.592033 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "9ca08b4c8b0c40e6a4c2e1a6fa873433"
format_stamp: "Formatted at 2026-09-25 10:25:08 on dist-test-slave-f9rl"
server_key: "9b5842d45f584b7931eaa67b4b84f89b"
server_key_iv: "aa6ea11541c519be5d99a2f4e6609a3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:08.592792 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "9ca08b4c8b0c40e6a4c2e1a6fa873433"
format_stamp: "Formatted at 2026-09-25 10:25:08 on dist-test-slave-f9rl"
server_key: "9b5842d45f584b7931eaa67b4b84f89b"
server_key_iv: "aa6ea11541c519be5d99a2f4e6609a3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:08.598158 13394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260925 10:25:08.601657 13409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:08.603075 13394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:25:08.603276 13394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "9ca08b4c8b0c40e6a4c2e1a6fa873433"
format_stamp: "Formatted at 2026-09-25 10:25:08 on dist-test-slave-f9rl"
server_key: "9b5842d45f584b7931eaa67b4b84f89b"
server_key_iv: "aa6ea11541c519be5d99a2f4e6609a3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:08.603497 13394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:08.621160 13394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:08.625300 13394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:08.625821 13394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:08.627295 13394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:08.629093 13394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:08.629187 13394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:08.629426 13394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:08.629523 13394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:08.668411 13394 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:46823
I20260925 10:25:08.668452 13521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:46823 every 8 connection(s)
I20260925 10:25:08.669787 13394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:08.674185 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13394
I20260925 10:25:08.674425 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:08.675225 13281 external_mini_cluster.cc:1428] Setting key b17268fe757261531bc08c5161aed2b1
Sep 25 10:25:08 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331908, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:08.695439 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:08.672473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:52421 (local address 127.12.248.126:41245)
0925 10:25:08.672926 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:08.672931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:08.674114 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 10:25:08.676760 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:08.676766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:08.676839 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:08.676959 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:08.679231 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.679939 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.682424 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.683157 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.683312 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:08.688361 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:08.688371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:08.688378 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:08.688426 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:08.692221 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:08.693031 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:08.693038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:08.693044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:08.693189 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:08.693736 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:08.693744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:08.693750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:08.694045 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:08.694163 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:08.694995 (+ 832us) server_negotiation.cc:299] Negotiation successful
0925 10:25:08.695174 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
I20260925 10:25:08.696813 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:08.673043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:41245 (local address 127.12.248.65:52421)
0925 10:25:08.673915 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:08.673966 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:08.676474 (+ 2508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:08.677304 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:08.677320 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:08.678140 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:08.679028 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:08.679060 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.680099 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.680108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:08.682284 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:08.682297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:08.683424 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:08.683433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:08.683781 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:08.684598 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:08.684639 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:08.688092 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:08.692376 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:08.692383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:08.692397 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:08.692885 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:08.693296 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:08.693299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:08.693301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:08.693599 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:08.694160 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:08.694166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:08.694580 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 10:25:08.694875 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 10:25:08.695288 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":272,"threads_started":1}
I20260925 10:25:08.698679 13522 heartbeater.cc:347] Connected to a master server at 127.12.248.126:41245
I20260925 10:25:08.699067 13522 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:08.699910 13522 heartbeater.cc:511] Master 127.12.248.126:41245 requested a full tablet report, sending...
I20260925 10:25:08.703071 13333 ts_manager.cc:194] Registered new tserver with Master: 9ca08b4c8b0c40e6a4c2e1a6fa873433 (127.12.248.65:46823)
I20260925 10:25:08.705374 13333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:52421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:08.721078 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:09.130990 13528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:09.131358 13528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:09.131549 13528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:09.139667 13528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:09.139822 13528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:09.140062 13528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:09.150549 13528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:09.151921 13528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:09.153308 13528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:09.168342 13533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:09.168783 13534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:09.168972 13536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:09.170336 13528 server_base.cc:1034] running on GCE node
I20260925 10:25:09.170971 13528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:09.171994 13528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:09.173283 13528 hybrid_clock.cc:648] HybridClock initialized: now 1790331909173164 us; error 59 us; skew 500 ppm
I20260925 10:25:09.173772 13528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:09.177337 13528 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:09.179035 13528 webserver.cc:460] Webserver started at http://127.12.248.66:39083/ using document root <none> and password file <none>
I20260925 10:25:09.179803 13528 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:09.179905 13528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:09.180294 13528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:09.183389 13528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "7e61c85a21db4033a1c1165d1a7492cd"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "977129af8ee25487d5554796ab6039d7"
server_key_iv: "d042d3e619611377d3237e91223ed991"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.184518 13528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "7e61c85a21db4033a1c1165d1a7492cd"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "977129af8ee25487d5554796ab6039d7"
server_key_iv: "d042d3e619611377d3237e91223ed991"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.191633 13528 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 10:25:09.196817 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.198547 13528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:25:09.198746 13528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "7e61c85a21db4033a1c1165d1a7492cd"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "977129af8ee25487d5554796ab6039d7"
server_key_iv: "d042d3e619611377d3237e91223ed991"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.198935 13528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:09.213155 13528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:09.218017 13528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:09.218334 13528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:09.219419 13528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:09.220901 13528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:09.220973 13528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.221102 13528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:09.221156 13528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.266649 13528 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:38249
I20260925 10:25:09.266685 13655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:38249 every 8 connection(s)
I20260925 10:25:09.268098 13528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:09.270381 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13528
I20260925 10:25:09.270624 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:09.271103 13281 external_mini_cluster.cc:1428] Setting key bd5b0385a4c87eadff7f6dbc814a13fd
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:09.292012 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.271203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:32841 (local address 127.12.248.126:41245)
0925 10:25:09.271743 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:09.271750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:09.273028 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 10:25:09.274883 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:09.274888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:09.274976 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:09.275094 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:09.276755 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.277728 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.279440 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.280319 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.280477 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.284111 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:09.284120 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:09.284129 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:09.284179 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:09.288662 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.289483 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.289490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.289495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.289696 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.290166 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.290182 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.290189 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.290635 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:09.290775 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:09.291507 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 10:25:09.291679 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:25:09.293125 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.271801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:41245 (local address 127.12.248.66:32841)
0925 10:25:09.272819 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:09.272875 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:09.274608 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:09.275340 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:09.275348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:09.275886 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:09.276454 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:09.276472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.277899 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.277911 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:09.279272 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:09.279286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.280571 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.280577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:09.280827 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.281651 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:09.281673 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:09.283839 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:09.288841 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:09.288848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:09.288862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:09.289314 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:09.289821 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:09.289825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:09.289827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:09.290043 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:09.290781 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:09.290787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:09.291134 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 10:25:09.291364 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 10:25:09.291859 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":425,"threads_started":1}
I20260925 10:25:09.294680 13656 heartbeater.cc:347] Connected to a master server at 127.12.248.126:41245
I20260925 10:25:09.295090 13656 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:09.296034 13656 heartbeater.cc:511] Master 127.12.248.126:41245 requested a full tablet report, sending...
I20260925 10:25:09.297748 13333 ts_manager.cc:194] Registered new tserver with Master: 7e61c85a21db4033a1c1165d1a7492cd (127.12.248.66:38249)
I20260925 10:25:09.299149 13333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:32841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:09.318048 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:09.705368 13662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:09.705811 13662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:09.705972 13662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:25:09.709824 13522 heartbeater.cc:503] Master 127.12.248.126:41245 was elected leader, sending a full tablet report...
W20260925 10:25:09.714306 13662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:09.714490 13662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:09.714727 13662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:09.725564 13662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:41245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:09.726943 13662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:09.728375 13662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:09.742808 13668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:09.742836 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:09.743587 13667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:09.744552 13662 server_base.cc:1034] running on GCE node
I20260925 10:25:09.745200 13662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:09.746284 13662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:09.747538 13662 hybrid_clock.cc:648] HybridClock initialized: now 1790331909747462 us; error 57 us; skew 500 ppm
I20260925 10:25:09.747871 13662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:09.751178 13662 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:09.752609 13662 webserver.cc:460] Webserver started at http://127.12.248.67:33545/ using document root <none> and password file <none>
I20260925 10:25:09.753289 13662 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:09.753388 13662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:09.753789 13662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:09.757107 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "bc15ccc9e5814dc99aa4ef1c514ae9d3"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "ebef3ab748474764d4e160254aacc1f6"
server_key_iv: "47e1d4983353e762ea3f973a718323d0"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.757905 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "bc15ccc9e5814dc99aa4ef1c514ae9d3"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "ebef3ab748474764d4e160254aacc1f6"
server_key_iv: "47e1d4983353e762ea3f973a718323d0"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.763176 13662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:25:09.766997 13677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.768396 13662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:25:09.768527 13662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "bc15ccc9e5814dc99aa4ef1c514ae9d3"
format_stamp: "Formatted at 2026-09-25 10:25:09 on dist-test-slave-f9rl"
server_key: "ebef3ab748474764d4e160254aacc1f6"
server_key_iv: "47e1d4983353e762ea3f973a718323d0"
server_key_version: "encryptionkey@0"
I20260925 10:25:09.768711 13662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:09.794456 13662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:09.799134 13662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:09.799499 13662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:09.800491 13662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:09.801831 13662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:09.801905 13662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.801988 13662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:09.802032 13662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:09.846011 13662 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:36503
I20260925 10:25:09.846063 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:36503 every 8 connection(s)
I20260925 10:25:09.847255 13662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:09.857393 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13662
I20260925 10:25:09.857652 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:09.858147 13281 external_mini_cluster.cc:1428] Setting key c1c5109d626d6d4efecb4a0f6086ebdc
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:09.868425 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.849845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:53263 (local address 127.12.248.126:41245)
0925 10:25:09.850271 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:09.850276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:09.851302 (+ 1026us) server_negotiation.cc:407] Connection header received
0925 10:25:09.853167 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:09.853174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:09.853280 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:09.853404 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:09.854894 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.855678 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.857303 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.857931 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.858069 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.861779 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:09.861785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:09.861790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:09.861834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:09.864922 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.865855 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.865862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.865866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.866012 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.866474 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.866483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.866485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.866795 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:09.866987 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:09.867711 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 10:25:09.867927 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":197,"threads_started":1}
I20260925 10:25:09.869562 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.850391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:41245 (local address 127.12.248.67:53263)
0925 10:25:09.851097 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:09.851135 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:09.852892 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:09.853670 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:09.853678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:09.854158 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:09.854724 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:09.854741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.855839 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.855846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:09.857151 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:09.857164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.858175 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.858184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:09.858451 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.859163 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:09.859186 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:09.861560 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:09.865075 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:09.865085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:09.865103 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:09.865700 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:09.866141 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:09.866146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:09.866150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:09.866346 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:09.866942 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:09.866951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:09.867313 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 10:25:09.867584 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 10:25:09.868149 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":289,"threads_started":1}
I20260925 10:25:09.871277 13790 heartbeater.cc:347] Connected to a master server at 127.12.248.126:41245
I20260925 10:25:09.871555 13790 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:09.872237 13790 heartbeater.cc:511] Master 127.12.248.126:41245 requested a full tablet report, sending...
I20260925 10:25:09.874027 13333 ts_manager.cc:194] Registered new tserver with Master: bc15ccc9e5814dc99aa4ef1c514ae9d3 (127.12.248.67:36503)
I20260925 10:25:09.874893 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:25:09.875223 13333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:53263
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:09.926060 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.908289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.12.248.65:46823)
0925 10:25:09.909035 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:09.909047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:09.909470 (+ 423us) server_negotiation.cc:407] Connection header received
0925 10:25:09.909707 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:09.909716 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:09.909955 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:09.910132 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:09.911780 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.912969 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.914373 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.915026 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.915176 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.919797 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:09.919808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:09.919823 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:09.919863 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:09.923066 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.923784 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.923792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.923794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.923961 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.924341 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.924344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.924346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.924741 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:09.925006 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:09.925354 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:25:09.925653 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":337,"threads_started":1}
W20260925 10:25:09.927501 13437 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36500
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:25:09 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:09.973822 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.958811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.12.248.65:46823)
0925 10:25:09.959029 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:09.959037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:09.959407 (+ 370us) server_negotiation.cc:407] Connection header received
0925 10:25:09.959681 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:09.959686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:09.959759 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:09.959867 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:09.961301 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.962092 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.963736 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.964226 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.964364 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.968548 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:09.968553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:09.968556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:09.968591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:09.971126 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.971899 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.971910 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.971913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.972035 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.972416 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.972422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.972426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.972679 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:09.972934 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:09.973237 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 10:25:09.973492 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:25:09.975615 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13394
I20260925 10:25:09.988565 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:09.976545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.12.248.65:46823)
0925 10:25:09.976803 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:09.976809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:09.977022 (+ 213us) server_negotiation.cc:407] Connection header received
0925 10:25:09.977132 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:09.977136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:09.977189 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:09.977268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:09.978535 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.979282 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.980497 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:09.980921 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:09.981014 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:09.983390 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:09.983394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:09.983398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:09.983426 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:09.986223 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.986821 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.986825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.986827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.987005 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:09.987330 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:09.987333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:09.987335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:09.987597 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:09.987788 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:09.988050 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 10:25:09.988205 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:25:09.989336 13517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:10.075213 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13394
I20260925 10:25:10.092721 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13528
Sep 25 10:25:10 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:10.109625 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:10.093842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.12.248.66:38249)
0925 10:25:10.094478 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:10.094485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:10.094513 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:25:10.094573 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:10.094580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:10.094804 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:10.094935 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:10.096077 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:10.097119 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:10.098385 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:10.099130 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:10.099229 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:10.103550 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:10.103564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:10.103582 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:10.103620 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:10.106781 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:10.107496 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:10.107504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:10.107506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:10.107628 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:10.108011 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:10.108014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:10.108016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:10.108370 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:10.108597 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:10.109016 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 10:25:10.109245 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":357,"threads_started":1}
I20260925 10:25:10.110944 13651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:10.194267 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13528
I20260925 10:25:10.211709 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13662
Sep 25 10:25:10 dist-test-slave-f9rl krb5kdc[13288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:10.229692 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:10.212649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.12.248.67:36503)
0925 10:25:10.213349 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:10.213357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:10.213387 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:25:10.213522 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:10.213530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:10.213761 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:10.213938 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:10.215227 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:10.216276 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:10.217600 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:10.218181 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:10.218343 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:10.222916 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:10.222929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:10.222944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:10.222986 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:10.226442 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:10.227359 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:10.227366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:10.227369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:10.227501 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:10.227917 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:10.227925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:10.227928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:10.228309 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:10.228663 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:10.228953 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:25:10.229243 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":363,"threads_started":1}
I20260925 10:25:10.230989 13785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:10.314464 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13662
I20260925 10:25:10.330502 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13301
I20260925 10:25:10.331696 13363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:10.434743 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13301
2026-09-25T10:25:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4162 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:10 dist-test-slave-f9rl krb5kdc[13812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:10 dist-test-slave-f9rl krb5kdc[13812](info): set up 2 sockets
Sep 25 10:25:10 dist-test-slave-f9rl krb5kdc[13812](info): commencing operation
krb5kdc: starting...
W20260925 10:25:11.530961 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:11 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:11.716817 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34347
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:34347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:12.043351 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:12.043663 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:12.043838 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:12.051254 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:12.051375 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:12.051486 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:12.051568 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:12.051610 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:12.061648 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:34347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:12.063099 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:12.064495 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:12.077586 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:12.077837 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:12.077633 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:12.077725 13825 server_base.cc:1034] running on GCE node
I20260925 10:25:12.078979 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:12.080361 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:12.081585 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790331912081532 us; error 47 us; skew 500 ppm
I20260925 10:25:12.081950 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:12 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:12.092993 13825 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:12.094689 13825 webserver.cc:460] Webserver started at http://127.12.248.126:43347/ using document root <none> and password file <none>
I20260925 10:25:12.095377 13825 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:12.095485 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:12.095757 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:12.098865 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "3e7ace4120404fb9abd3f977d345ecb8"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "02d4777192d4751dc6d2d91b6b6548b7"
server_key_iv: "dabcaba868aa3a1d7539dbbe57f75dd6"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.099583 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "3e7ace4120404fb9abd3f977d345ecb8"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "02d4777192d4751dc6d2d91b6b6548b7"
server_key_iv: "dabcaba868aa3a1d7539dbbe57f75dd6"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.104835 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:12.108373 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:12.109805 13825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:12.109952 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "3e7ace4120404fb9abd3f977d345ecb8"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "02d4777192d4751dc6d2d91b6b6548b7"
server_key_iv: "dabcaba868aa3a1d7539dbbe57f75dd6"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.110127 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:12.121891 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:12.126106 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:12.126376 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:12.145203 13825 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:34347
I20260925 10:25:12.145308 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:34347 every 8 connection(s)
I20260925 10:25:12.146752 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:12.151083 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:12.154354 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13825
I20260925 10:25:12.154614 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:12.155542 13281 external_mini_cluster.cc:1428] Setting key 28fe5d5bb8fe5f37ecf8f331414f629d
I20260925 10:25:12.165238 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8: Bootstrap starting.
I20260925 10:25:12.169682 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:12.171149 13892 log.cc:824] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:25:12 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:12.174839 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8: No bootstrap required, opened a new log
I20260925 10:25:12.180593 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:12.157903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.12.248.126:34347)
0925 10:25:12.158724 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:12.158737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:12.158778 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:25:12.160462 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:12.160484 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:12.160912 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:12.161262 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:12.163327 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.164651 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.166583 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.167327 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.167459 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:12.172889 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:12.172897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:12.172912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:12.172951 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:12.176178 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:12.177005 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:12.177012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:12.177019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:12.177190 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:12.177703 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:12.177707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:12.177709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:12.178265 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:12.178628 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:12.179099 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:25:12.179538 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":386,"threads_started":1}
I20260925 10:25:12.184537 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } }
I20260925 10:25:12.184976 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:12.185099 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7ace4120404fb9abd3f977d345ecb8, State: Initialized, Role: FOLLOWER
I20260925 10:25:12.185746 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } }
I20260925 10:25:12.186036 13892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:12.186201 13892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:12.186393 13892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:12.188906 13892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } }
I20260925 10:25:12.189551 13892 leader_election.cc:302] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e7ace4120404fb9abd3f977d345ecb8; no voters:
I20260925 10:25:12.190433 13892 leader_election.cc:288] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:12.190507 13897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:12.191264 13897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7ace4120404fb9abd3f977d345ecb8, State: Running, Role: LEADER
I20260925 10:25:12.191942 13897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } }
I20260925 10:25:12.193014 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:12.194960 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e7ace4120404fb9abd3f977d345ecb8. Latest consensus state: current_term: 1 leader_uuid: "3e7ace4120404fb9abd3f977d345ecb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } } }
I20260925 10:25:12.195492 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:12.196386 13899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e7ace4120404fb9abd3f977d345ecb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7ace4120404fb9abd3f977d345ecb8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34347 } } }
I20260925 10:25:12.196774 13899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:12.197227 13902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:12.202432 13902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:12.212327 13902 catalog_manager.cc:1347] Generated new cluster ID: aeb14c17fa2940c4bbd8b4e2e52e7323
I20260925 10:25:12.212455 13902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:12.226002 13902 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:12.227663 13902 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:12.235400 13902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e7ace4120404fb9abd3f977d345ecb8: Generated new TSK 0
I20260925 10:25:12.236259 13902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:12.286293 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:12.633749 13918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:12.634027 13918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:12.634203 13918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:12.641330 13918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:12.641500 13918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:12.641649 13918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:12.651532 13918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:12.652805 13918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:12.654184 13918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:12.667654 13923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:12.667716 13924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:12.667692 13926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:12.668553 13918 server_base.cc:1034] running on GCE node
I20260925 10:25:12.669124 13918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:12.670092 13918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:12.671298 13918 hybrid_clock.cc:648] HybridClock initialized: now 1790331912671260 us; error 31 us; skew 500 ppm
I20260925 10:25:12.671576 13918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:12 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:12.674811 13918 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:12.676322 13918 webserver.cc:460] Webserver started at http://127.12.248.65:39347/ using document root <none> and password file <none>
I20260925 10:25:12.677088 13918 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:12.677244 13918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:12.677642 13918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:12.681051 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "7e295dd9d2f5411e9e802dd27b93a8aa"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "cb135843114f60136a2cb49770f37308"
server_key_iv: "78f963002c07ad94f976a1e7dcd407c9"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.681931 13918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "7e295dd9d2f5411e9e802dd27b93a8aa"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "cb135843114f60136a2cb49770f37308"
server_key_iv: "78f963002c07ad94f976a1e7dcd407c9"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.687062 13918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:25:12.690681 13933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:12.692037 13918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:25:12.692219 13918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "7e295dd9d2f5411e9e802dd27b93a8aa"
format_stamp: "Formatted at 2026-09-25 10:25:12 on dist-test-slave-f9rl"
server_key: "cb135843114f60136a2cb49770f37308"
server_key_iv: "78f963002c07ad94f976a1e7dcd407c9"
server_key_version: "encryptionkey@0"
I20260925 10:25:12.692435 13918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:12.708359 13918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:12.712834 13918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:12.713163 13918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:12.714349 13918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:12.715662 13918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:12.715744 13918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:12.715878 13918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:12.715941 13918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:12.748026 13918 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:44593
I20260925 10:25:12.748062 14045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:44593 every 8 connection(s)
I20260925 10:25:12.749513 13918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:12.756848 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 13918
I20260925 10:25:12.757077 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:12.757803 13281 external_mini_cluster.cc:1428] Setting key e13972693b654a3940069ebd5ad95922
Sep 25 10:25:12 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331912, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:12.771414 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:12.752176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:45021 (local address 127.12.248.126:34347)
0925 10:25:12.752618 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:12.752623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:12.753618 (+ 995us) server_negotiation.cc:407] Connection header received
0925 10:25:12.755454 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:12.755459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:12.755547 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:12.755670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:12.757202 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.758104 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.759950 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.760350 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.760492 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:12.764170 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:12.764175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:12.764178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:12.764210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:12.767770 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:12.768501 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:12.768509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:12.768515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:12.768660 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:12.769081 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:12.769089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:12.769094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:12.769518 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:12.769680 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:12.770387 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 10:25:12.770658 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260925 10:25:12.771982 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:12.752736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34347 (local address 127.12.248.65:45021)
0925 10:25:12.753406 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:12.753490 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:12.755174 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:12.755917 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:12.755925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:12.756410 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:12.757033 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:12.757053 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.758270 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.758278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:12.759808 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:12.759820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:12.760577 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:12.760585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:12.760844 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:12.761588 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:12.761610 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:12.763947 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:12.767921 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:12.767929 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:12.767943 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:12.768371 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:12.768779 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:12.768782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:12.768784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:12.768969 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:12.769647 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:12.769653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:12.770000 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 10:25:12.770237 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 10:25:12.770631 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":262,"threads_started":1}
I20260925 10:25:12.773705 14046 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34347
I20260925 10:25:12.774070 14046 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:12.775033 14046 heartbeater.cc:511] Master 127.12.248.126:34347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:12.777372 13857 ts_manager.cc:194] Registered new tserver with Master: 7e295dd9d2f5411e9e802dd27b93a8aa (127.12.248.65:44593)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
I20260925 10:25:12.779399 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:45021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:12.801144 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:13.129086 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:13.129451 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:13.129630 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:13.137128 14052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:13.137301 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:13.137557 14052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:13.147899 14052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:13.149227 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:13.150662 14052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:13.164693 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:13.164798 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:13.164803 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:13.165675 14052 server_base.cc:1034] running on GCE node
I20260925 10:25:13.166290 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:13.167294 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:13.168499 14052 hybrid_clock.cc:648] HybridClock initialized: now 1790331913168439 us; error 51 us; skew 500 ppm
I20260925 10:25:13.168830 14052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:13 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:13.172222 14052 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:13.173907 14052 webserver.cc:460] Webserver started at http://127.12.248.66:37793/ using document root <none> and password file <none>
I20260925 10:25:13.174631 14052 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:13.174777 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:13.175112 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:13.178313 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "18be6b424e774958bcb3f8898709f52d"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a7093ff58afbff2c58ec71b9d065927"
server_key_iv: "5132c70b6628c6fbe38e276491f3f23c"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.179136 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "18be6b424e774958bcb3f8898709f52d"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a7093ff58afbff2c58ec71b9d065927"
server_key_iv: "5132c70b6628c6fbe38e276491f3f23c"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.184393 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:13.187918 14067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.189225 14052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:13.189397 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "18be6b424e774958bcb3f8898709f52d"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a7093ff58afbff2c58ec71b9d065927"
server_key_iv: "5132c70b6628c6fbe38e276491f3f23c"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.189656 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:13.207440 14052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:13.211517 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:13.211817 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:13.212797 14052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:13.214260 14052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:13.214334 14052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.214484 14052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:13.214534 14052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.250236 14052 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:45525
I20260925 10:25:13.250283 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:45525 every 8 connection(s)
I20260925 10:25:13.251565 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:13.258590 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14052
I20260925 10:25:13.258808 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:13.259258 13281 external_mini_cluster.cc:1428] Setting key 205ab9d5728595d8efa4ed31b72c730d
Sep 25 10:25:13 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:13.271826 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.254091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:43567 (local address 127.12.248.126:34347)
0925 10:25:13.254275 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.254287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.255554 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 10:25:13.257422 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.257428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.257552 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.257671 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:13.259291 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.260169 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.261817 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.262251 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.262390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.266149 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:13.266154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:13.266157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:13.266189 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:13.268785 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.269540 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.269548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.269551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.269679 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.270114 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.270119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.270124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.270395 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:13.270554 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.271261 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.271486 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:25:13.273224 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.254548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34347 (local address 127.12.248.66:43567)
0925 10:25:13.255369 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.255409 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.257141 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.257922 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.257931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.258425 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:13.259082 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.259100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.260340 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.260344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.261650 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.261662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.262499 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.262507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.262739 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.263439 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:13.263461 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:13.265906 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:13.268933 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:13.268941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:13.268959 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:13.269365 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:13.269797 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:13.269802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:13.269806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:13.269990 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:13.270527 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:13.270534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:13.270883 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.271157 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.271689 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":391,"threads_started":1}
I20260925 10:25:13.274788 14180 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34347
I20260925 10:25:13.275144 14180 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:13.276079 14180 heartbeater.cc:511] Master 127.12.248.126:34347 requested a full tablet report, sending...
I20260925 10:25:13.277678 13857 ts_manager.cc:194] Registered new tserver with Master: 18be6b424e774958bcb3f8898709f52d (127.12.248.66:45525)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:25:13.278654 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:43567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:13.301280 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:13.626245 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:13.626515 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:13.626677 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:13.634585 14185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:13.634729 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:13.634923 14185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:13.644722 14185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34347
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:13.646181 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:13.647562 14185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:13.661003 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:13.660912 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:13.660981 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:13.662139 14185 server_base.cc:1034] running on GCE node
I20260925 10:25:13.662770 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:13.663792 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:13.665012 14185 hybrid_clock.cc:648] HybridClock initialized: now 1790331913664967 us; error 40 us; skew 500 ppm
I20260925 10:25:13.665305 14185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:13 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:13.668448 14185 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:13.669997 14185 webserver.cc:460] Webserver started at http://127.12.248.67:36833/ using document root <none> and password file <none>
I20260925 10:25:13.670674 14185 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:13.670775 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:13.671043 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:13.674063 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "43a543c1cf524694bc3e7effef53af3b"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a6a27964bc01c5479ba169cbc7695f3"
server_key_iv: "88080efa2d4eb59cfc7ec52988440b62"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.674845 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "43a543c1cf524694bc3e7effef53af3b"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a6a27964bc01c5479ba169cbc7695f3"
server_key_iv: "88080efa2d4eb59cfc7ec52988440b62"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.680008 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:13.683442 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.684778 14185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:13.684939 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "43a543c1cf524694bc3e7effef53af3b"
format_stamp: "Formatted at 2026-09-25 10:25:13 on dist-test-slave-f9rl"
server_key: "0a6a27964bc01c5479ba169cbc7695f3"
server_key_iv: "88080efa2d4eb59cfc7ec52988440b62"
server_key_version: "encryptionkey@0"
I20260925 10:25:13.685150 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:13.705421 14185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:13.710158 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:13.710475 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:13.711493 14185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:13.712862 14185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:13.712971 14185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.713140 14185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:13.713218 14185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:13.747761 14185 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:44241
I20260925 10:25:13.747838 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:44241 every 8 connection(s)
I20260925 10:25:13.749385 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:13.758219 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14185
I20260925 10:25:13.758424 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:13.758988 13281 external_mini_cluster.cc:1428] Setting key 20400dbc61ea367e53903cb6965cbfd9
Sep 25 10:25:13 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331913, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:13.769670 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.752168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:40231 (local address 127.12.248.126:34347)
0925 10:25:13.752351 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.752355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.753824 (+ 1469us) server_negotiation.cc:407] Connection header received
0925 10:25:13.755619 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.755625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.755746 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.755867 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:13.757317 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.758334 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.760056 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.760712 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.760851 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.764417 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:13.764421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:13.764425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:13.764455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:13.766885 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.767648 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.767652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.767654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.767774 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.768159 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.768162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.768164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.768416 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:13.768521 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.769175 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.769344 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:25:13.770673 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.752677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34347 (local address 127.12.248.67:40231)
0925 10:25:13.753626 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.753677 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.755341 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.756095 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.756104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.756584 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:13.757133 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.757150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.758517 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.758525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.759896 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.759913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.760945 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.760951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.761249 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.762039 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:13.762076 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:13.764217 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:13.767032 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:13.767049 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:13.767068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:13.767545 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:13.767875 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:13.767878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:13.767880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:13.768061 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:13.768548 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:13.768554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:13.768856 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.769109 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.769598 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":353,"threads_started":1}
I20260925 10:25:13.772243 14313 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34347
I20260925 10:25:13.772506 14313 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:13.773392 14313 heartbeater.cc:511] Master 127.12.248.126:34347 requested a full tablet report, sending...
I20260925 10:25:13.774979 13857 ts_manager.cc:194] Registered new tserver with Master: 43a543c1cf524694bc3e7effef53af3b (127.12.248.67:44241)
I20260925 10:25:13.775782 13857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:40231
I20260925 10:25:13.782553 14046 heartbeater.cc:503] Master 127.12.248.126:34347 was elected leader, sending a full tablet report...
I20260925 10:25:13.786825 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:25:13.806537 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.793594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.12.248.126:34347)
0925 10:25:13.793775 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.793780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.794394 (+ 614us) server_negotiation.cc:407] Connection header received
0925 10:25:13.794566 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.794573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.794645 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.794729 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:13.796346 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.797166 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.798476 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.798980 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.799075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.801517 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:13.801522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:13.801525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:13.801555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:13.804154 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.804734 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.804737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.804739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.804859 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:13.805177 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:13.805180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:13.805182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:13.805603 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:13.805807 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.806100 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.806272 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:25:13.813141 13857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:13.820338 13857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:13.850296 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.833561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:44241 (local address 127.0.0.1:49344)
0925 10:25:13.835172 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.835233 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.835518 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.837172 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.837184 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.837254 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:13.837851 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.837869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.841293 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.841303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.844672 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.844700 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.849238 (+ 4538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.849244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.849576 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.849662 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.849774 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.849927 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":22,"thread_start_us":13950,"threads_started":1}
I20260925 10:25:13.851764 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.833869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.12.248.67:44241)
0925 10:25:13.836340 (+ 2471us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.836351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.836392 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:25:13.836517 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.836529 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.836861 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.838122 (+ 1261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:13.838489 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.841036 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.844992 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.847204 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.850810 (+ 3606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.851160 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.851240 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.851432 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2246,"thread_start_us":2032,"threads_started":1}
I20260925 10:25:13.856400 14248 tablet_service.cc:1467] Processing CreateTablet for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8 (DEFAULT_TABLE table=test-table [id=a45f3bf292f44c079a062b01403c77f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:13.857372 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.833935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.12.248.66:45525)
0925 10:25:13.834503 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.834516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.847985 (+ 13469us) server_negotiation.cc:407] Connection header received
0925 10:25:13.848343 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.848354 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.848665 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.848863 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:13.849761 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.851307 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.854786 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.856205 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.856355 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.856567 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.856879 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.857060 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260925 10:25:13.857875 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.833727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:45525 (local address 127.0.0.1:43548)
0925 10:25:13.847830 (+ 14103us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.847853 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.848119 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.848971 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.848977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.849007 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:13.849548 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.849560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.851464 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.851469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.854590 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.854606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.856431 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.856438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.856650 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.856689 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.856783 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.856982 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13960,"mutex_wait_us":4,"thread_start_us":963,"threads_started":1}
I20260925 10:25:13.858242 14248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:13.858801 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.836272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:44593 (local address 127.0.0.1:51824)
0925 10:25:13.848594 (+ 12322us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.848617 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.848773 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.850352 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.850359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.850495 (+ 136us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:13.850872 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.850901 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.853227 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.853234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.855777 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.855804 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.858066 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.858074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.858274 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.858335 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.858412 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.858527 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12139,"thread_start_us":12593,"threads_started":1}
I20260925 10:25:13.860400 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.836453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.12.248.65:44593)
0925 10:25:13.849710 (+ 13257us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.849720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.849746 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:25:13.849844 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.849855 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.850131 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.850333 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:13.851320 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.853062 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.855998 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.857755 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.859413 (+ 1658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.859700 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.859777 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.859984 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13052,"thread_start_us":12868,"threads_started":1}
I20260925 10:25:13.862121 14115 tablet_service.cc:1467] Processing CreateTablet for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8 (DEFAULT_TABLE table=test-table [id=a45f3bf292f44c079a062b01403c77f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:13.863615 14115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:13.863704 13981 tablet_service.cc:1467] Processing CreateTablet for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8 (DEFAULT_TABLE table=test-table [id=a45f3bf292f44c079a062b01403c77f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:13.865181 13981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:13.874292 14331 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Bootstrap starting.
I20260925 10:25:13.879324 14331 tablet_bootstrap.cc:654] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:13.881131 14331 log.cc:824] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:13.882109 14332 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Bootstrap starting.
I20260925 10:25:13.882938 14333 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Bootstrap starting.
I20260925 10:25:13.886234 14331 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: No bootstrap required, opened a new log
I20260925 10:25:13.886610 14331 ts_tablet_manager.cc:1397] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260925 10:25:13.887180 14332 tablet_bootstrap.cc:654] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:13.887321 14333 tablet_bootstrap.cc:654] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:13.888572 14333 log.cc:824] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:13.888620 14332 log.cc:824] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:13.892292 14333 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: No bootstrap required, opened a new log
I20260925 10:25:13.892290 14332 tablet_bootstrap.cc:492] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: No bootstrap required, opened a new log
I20260925 10:25:13.892582 14333 ts_tablet_manager.cc:1397] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 10:25:13.892582 14332 ts_tablet_manager.cc:1397] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260925 10:25:13.899773 14331 raft_consensus.cc:348] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.900388 14331 raft_consensus.cc:374] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:13.900503 14331 raft_consensus.cc:729] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a543c1cf524694bc3e7effef53af3b, State: Initialized, Role: FOLLOWER
I20260925 10:25:13.901235 14331 consensus_queue.cc:260] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.902587 14313 heartbeater.cc:503] Master 127.12.248.126:34347 was elected leader, sending a full tablet report...
I20260925 10:25:13.903455 14331 ts_tablet_manager.cc:1428] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260925 10:25:13.904781 14333 raft_consensus.cc:348] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.905369 14333 raft_consensus.cc:374] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:13.905494 14333 raft_consensus.cc:729] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e295dd9d2f5411e9e802dd27b93a8aa, State: Initialized, Role: FOLLOWER
I20260925 10:25:13.906240 14333 consensus_queue.cc:260] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.906160 14332 raft_consensus.cc:348] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.906608 14332 raft_consensus.cc:374] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:13.906729 14332 raft_consensus.cc:729] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18be6b424e774958bcb3f8898709f52d, State: Initialized, Role: FOLLOWER
I20260925 10:25:13.907500 14332 consensus_queue.cc:260] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.909261 14333 ts_tablet_manager.cc:1428] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260925 10:25:13.910576 14180 heartbeater.cc:503] Master 127.12.248.126:34347 was elected leader, sending a full tablet report...
I20260925 10:25:13.911192 14332 ts_tablet_manager.cc:1428] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260925 10:25:13.952615 14337 raft_consensus.cc:482] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:13.952975 14337 raft_consensus.cc:504] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.955394 14337 leader_election.cc:288] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18be6b424e774958bcb3f8898709f52d (127.12.248.66:45525), 7e295dd9d2f5411e9e802dd27b93a8aa (127.12.248.65:44593)
I20260925 10:25:13.961692 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.955854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:45525 (local address 127.12.248.67:46337)
0925 10:25:13.956151 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.956189 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.956351 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.956672 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.956680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.956705 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:13.956968 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.956979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.958141 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.958146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.959546 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.959561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.960911 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.960917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.961108 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.961132 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.961249 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.961377 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 10:25:13.961696 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.955761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:46337 (local address 127.12.248.66:45525)
0925 10:25:13.955969 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.955974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.956285 (+ 311us) server_negotiation.cc:407] Connection header received
0925 10:25:13.956492 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.956497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.956563 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.956658 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:13.957116 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.957991 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.959756 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.960761 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.960921 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.960995 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.961274 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.961407 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:25:13.962982 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.956187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:44593 (local address 127.12.248.67:56169)
0925 10:25:13.957188 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:13.957202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:13.957346 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:13.957692 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:13.957696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:13.957716 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:13.958083 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.958095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.959352 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.959357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.960808 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:13.960820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.962443 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.962448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:13.962562 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.962585 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:25:13.962659 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:25:13.962758 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":625,"threads_started":1}
I20260925 10:25:13.962865 14135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b927ca65b04a78bb4cfefcf5b5d4a8" candidate_uuid: "43a543c1cf524694bc3e7effef53af3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18be6b424e774958bcb3f8898709f52d" is_pre_election: true
I20260925 10:25:13.963084 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:13.955726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:56169 (local address 127.12.248.65:44593)
0925 10:25:13.955970 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:13.955976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:13.957297 (+ 1321us) server_negotiation.cc:407] Connection header received
0925 10:25:13.957517 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:13.957521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:13.957581 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:13.957679 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:13.958291 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.959085 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.961005 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:13.962239 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:13.962437 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:13.962515 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:13.962710 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 10:25:13.962844 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:25:13.963670 14135 raft_consensus.cc:2454] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a543c1cf524694bc3e7effef53af3b in term 0.
I20260925 10:25:13.964076 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b927ca65b04a78bb4cfefcf5b5d4a8" candidate_uuid: "43a543c1cf524694bc3e7effef53af3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" is_pre_election: true
I20260925 10:25:13.964428 14203 leader_election.cc:302] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18be6b424e774958bcb3f8898709f52d, 43a543c1cf524694bc3e7effef53af3b; no voters:
I20260925 10:25:13.964771 14001 raft_consensus.cc:2454] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a543c1cf524694bc3e7effef53af3b in term 0.
I20260925 10:25:13.964843 14337 raft_consensus.cc:2781] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:13.965087 14337 raft_consensus.cc:482] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:13.965306 14337 raft_consensus.cc:3037] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:13.968523 14337 raft_consensus.cc:504] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.969262 14337 leader_election.cc:288] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [CANDIDATE]: Term 1 election: Requested vote from peers 18be6b424e774958bcb3f8898709f52d (127.12.248.66:45525), 7e295dd9d2f5411e9e802dd27b93a8aa (127.12.248.65:44593)
I20260925 10:25:13.969709 14135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b927ca65b04a78bb4cfefcf5b5d4a8" candidate_uuid: "43a543c1cf524694bc3e7effef53af3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18be6b424e774958bcb3f8898709f52d"
I20260925 10:25:13.969835 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b927ca65b04a78bb4cfefcf5b5d4a8" candidate_uuid: "43a543c1cf524694bc3e7effef53af3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa"
I20260925 10:25:13.969944 14135 raft_consensus.cc:3037] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:13.970142 14001 raft_consensus.cc:3037] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:13.973783 14135 raft_consensus.cc:2454] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a543c1cf524694bc3e7effef53af3b in term 1.
I20260925 10:25:13.974212 14203 leader_election.cc:302] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18be6b424e774958bcb3f8898709f52d, 43a543c1cf524694bc3e7effef53af3b; no voters:
I20260925 10:25:13.974371 14001 raft_consensus.cc:2454] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a543c1cf524694bc3e7effef53af3b in term 1.
I20260925 10:25:13.974609 14337 raft_consensus.cc:2781] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:13.975097 14337 raft_consensus.cc:686] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [term 1 LEADER]: Becoming Leader. State: Replica: 43a543c1cf524694bc3e7effef53af3b, State: Running, Role: LEADER
I20260925 10:25:13.975636 14337 consensus_queue.cc:237] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } }
I20260925 10:25:13.981353 13856 catalog_manager.cc:5495] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 43a543c1cf524694bc3e7effef53af3b (127.12.248.67). New cstate: current_term: 1 leader_uuid: "43a543c1cf524694bc3e7effef53af3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a543c1cf524694bc3e7effef53af3b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18be6b424e774958bcb3f8898709f52d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e295dd9d2f5411e9e802dd27b93a8aa" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44593 } health_report { overall_health: UNKNOWN } } }
W20260925 10:25:14.001242 14314 tablet.cc:2369] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 43a543c1cf524694bc3e7effef53af3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:25:14.001834 14047 tablet.cc:2369] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 7e295dd9d2f5411e9e802dd27b93a8aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:25:14.003540 14181 tablet.cc:2369] T 65b927ca65b04a78bb4cfefcf5b5d4a8 P 18be6b424e774958bcb3f8898709f52d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:14.021836 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.005304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.12.248.65:44593)
0925 10:25:14.005538 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.005542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.006276 (+ 734us) server_negotiation.cc:407] Connection header received
0925 10:25:14.006500 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.006509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.006579 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.006723 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.008497 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.009256 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.010605 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.011197 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.011340 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.015606 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.015620 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.015637 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.015672 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.018710 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.019428 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.019435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.019438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.019573 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.020022 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.020027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.020029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.020458 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.020860 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.021183 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.021473 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:14.070101 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.054527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.12.248.65:44593)
0925 10:25:14.054786 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.054793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.055054 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:25:14.055335 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.055339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.055403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.055509 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.056929 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.057832 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.059456 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.059879 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.060023 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.064352 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.064357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.064361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.064402 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.067131 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.067788 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.067795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.067798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.067985 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.068371 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.068377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.068380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.068682 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.068981 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.069489 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.069766 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:14.118419 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.104252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.12.248.65:44593)
0925 10:25:14.104487 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.104496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.104678 (+ 182us) server_negotiation.cc:407] Connection header received
0925 10:25:14.104833 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.104838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.104901 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.104985 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.106468 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.107179 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.108461 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.108908 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.108999 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.113354 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.113359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.113363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.113400 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.115950 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.116595 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.116599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.116601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.116735 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.117091 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.117094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.117096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.117403 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.117691 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.117979 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.118165 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 10:25:14.120424 13961 tablet_service.cc:2908] Rejecting scan request for tablet 65b927ca65b04a78bb4cfefcf5b5d4a8: Uninitialized: clean time has not yet been initialized
I20260925 10:25:14.123744 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13918
I20260925 10:25:14.137166 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.124659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.12.248.65:44593)
0925 10:25:14.124850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.124857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.125143 (+ 286us) server_negotiation.cc:407] Connection header received
0925 10:25:14.125342 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.125348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.125426 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.125562 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.126777 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.127453 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.128746 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.129132 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.129294 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.131748 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.131753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.131756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.131786 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.134468 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.135139 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.135144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.135148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.135316 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.135685 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.135693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.135697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.136001 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.136438 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.136633 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.136844 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:25:14.137936 14041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:14.230089 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13918
I20260925 10:25:14.247751 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14052
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:14.266927 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.248644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.12.248.66:45525)
0925 10:25:14.248824 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.248829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.248848 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:25:14.248993 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.248997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.249059 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.249220 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.250892 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.251719 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.253665 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.254379 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.254505 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.259608 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.259621 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.259639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.259679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.263897 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.264773 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.264783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.264787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.264944 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.265403 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.265408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.265412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.265962 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.266313 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.266482 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.266664 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:14.267635 14175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:14.357831 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14052
I20260925 10:25:14.375031 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14185
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:14.390981 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:14.375795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.12.248.67:44241)
0925 10:25:14.376274 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:14.376278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:14.376294 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:25:14.376349 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:14.376353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:14.376439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:14.376565 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:14.378073 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.378830 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.380135 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:14.380540 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:14.380637 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:14.384927 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:14.384943 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:14.384960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:14.384995 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:14.387942 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.388677 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.388685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.388688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.388822 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:14.389225 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:14.389232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:14.389240 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:14.389809 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:14.390059 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:14.390389 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 10:25:14.390588 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260925 10:25:14.391809 14308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:14.481410 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14185
I20260925 10:25:14.494848 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 13825
I20260925 10:25:14.495975 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:14.602980 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 13825
2026-09-25T10:25:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4169 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[14361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[14361](info): set up 2 sockets
Sep 25 10:25:14 dist-test-slave-f9rl krb5kdc[14361](info): commencing operation
krb5kdc: starting...
W20260925 10:25:15.691045 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:15 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:15.874724 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44581
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:16.217257 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:16.217665 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:16.217774 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:16.225020 14374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:16.225132 14374 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:25:16.225173 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:16.225236 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:16.225286 14374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:16.225343 14374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:16.235497 14374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44581
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:16.236755 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:16.238415 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:16.252216 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:16.252329 14380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:16.252529 14382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:16.253079 14374 server_base.cc:1034] running on GCE node
I20260925 10:25:16.253986 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:16.255494 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:16.256776 14374 hybrid_clock.cc:648] HybridClock initialized: now 1790331916256710 us; error 56 us; skew 500 ppm
I20260925 10:25:16.257133 14374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:16 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:16.260491 14374 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:16.262135 14374 webserver.cc:460] Webserver started at http://127.12.248.126:39063/ using document root <none> and password file <none>
I20260925 10:25:16.262851 14374 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:16.262948 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:16.263242 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:16.266497 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "4eeda86f85364cec8e98ebc9d89ec0fe"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "d6da762968f8c379b5d2695691dff0c5"
server_key_iv: "05841a41c03bc7f54ad0ab0735708663"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.267282 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "4eeda86f85364cec8e98ebc9d89ec0fe"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "d6da762968f8c379b5d2695691dff0c5"
server_key_iv: "05841a41c03bc7f54ad0ab0735708663"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.272922 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:16.276355 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:16.277710 14374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:16.277904 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "4eeda86f85364cec8e98ebc9d89ec0fe"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "d6da762968f8c379b5d2695691dff0c5"
server_key_iv: "05841a41c03bc7f54ad0ab0735708663"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.278116 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:16.297334 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:16.301751 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:16.302044 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:16.324137 14374 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:44581
I20260925 10:25:16.324146 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:44581 every 8 connection(s)
I20260925 10:25:16.325568 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:16.329288 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:16.332770 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14374
I20260925 10:25:16.333020 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:16.333885 13281 external_mini_cluster.cc:1428] Setting key fcf05c0342d2e9539ff8437cbbf5daef
I20260925 10:25:16.342871 14451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Bootstrap starting.
I20260925 10:25:16.347396 14451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:16.348997 14451 log.cc:824] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:25:16 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:16.352346 14451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: No bootstrap required, opened a new log
I20260925 10:25:16.357342 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:16.335958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.12.248.126:44581)
0925 10:25:16.336772 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:16.336784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:16.336825 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:25:16.338541 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:16.338565 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:16.338993 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:16.339339 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:16.341196 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.342551 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.344243 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.344923 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.345051 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:16.350327 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:16.350335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:16.350353 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:16.350387 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:16.353359 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.354067 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.354078 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.354084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.354254 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.354626 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.354632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.354635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.355125 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:16.355475 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:16.355899 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 10:25:16.356332 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":377,"threads_started":1}
I20260925 10:25:16.360361 14451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } }
I20260925 10:25:16.360716 14451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:16.360832 14451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eeda86f85364cec8e98ebc9d89ec0fe, State: Initialized, Role: FOLLOWER
I20260925 10:25:16.361370 14451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } }
I20260925 10:25:16.361670 14451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:16.361809 14451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:16.361984 14451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:16.364142 14451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } }
I20260925 10:25:16.364634 14451 leader_election.cc:302] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eeda86f85364cec8e98ebc9d89ec0fe; no voters:
I20260925 10:25:16.365197 14451 leader_election.cc:288] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:16.365371 14456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:16.366674 14456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [term 1 LEADER]: Becoming Leader. State: Replica: 4eeda86f85364cec8e98ebc9d89ec0fe, State: Running, Role: LEADER
I20260925 10:25:16.367172 14456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } }
I20260925 10:25:16.367463 14451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:16.369961 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eeda86f85364cec8e98ebc9d89ec0fe. Latest consensus state: current_term: 1 leader_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } } }
I20260925 10:25:16.370422 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:16.372779 14463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:16.372742 14457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eeda86f85364cec8e98ebc9d89ec0fe" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44581 } } }
I20260925 10:25:16.373114 14457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:16.378094 14463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:16.387226 14463 catalog_manager.cc:1347] Generated new cluster ID: 67ca1dbdf6e14d48a78f835f5e441862
I20260925 10:25:16.387389 14463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:16.408198 14463 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:16.409593 14463 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:16.417179 14463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Generated new TSK 0
I20260925 10:25:16.418262 14463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:16.475348 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44581
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:16.801158 14477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:16.801486 14477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:16.801662 14477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:16.808815 14477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:16.808976 14477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:16.809135 14477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:16.809278 14477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:16.818997 14477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:16.820326 14477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:16.821705 14477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:16.834786 14485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:16.834796 14483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:16.834894 14482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:16.835760 14477 server_base.cc:1034] running on GCE node
I20260925 10:25:16.836503 14477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:16.837628 14477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:16.838927 14477 hybrid_clock.cc:648] HybridClock initialized: now 1790331916838820 us; error 56 us; skew 500 ppm
I20260925 10:25:16.839347 14477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:16 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:16.842643 14477 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:16.844081 14477 webserver.cc:460] Webserver started at http://127.12.248.65:40009/ using document root <none> and password file <none>
I20260925 10:25:16.844995 14477 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:16.845147 14477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:16.845618 14477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:16.848912 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "929c64ff70954b24b644b5b0c1f15f9e"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "4da3eeeed2fb86649ee5224bde53ef9f"
server_key_iv: "814f938653eda4aaa7dd5d4f5241339e"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.849864 14477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "929c64ff70954b24b644b5b0c1f15f9e"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "4da3eeeed2fb86649ee5224bde53ef9f"
server_key_iv: "814f938653eda4aaa7dd5d4f5241339e"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.855556 14477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:16.860556 14492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:16.862188 14477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:16.862417 14477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "929c64ff70954b24b644b5b0c1f15f9e"
format_stamp: "Formatted at 2026-09-25 10:25:16 on dist-test-slave-f9rl"
server_key: "4da3eeeed2fb86649ee5224bde53ef9f"
server_key_iv: "814f938653eda4aaa7dd5d4f5241339e"
server_key_version: "encryptionkey@0"
I20260925 10:25:16.862756 14477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:16.878141 14477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:16.882380 14477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:16.882723 14477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:16.885504 14477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:16.885615 14477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:16.885814 14477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:16.886046 14477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:25:16 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:16.913609 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:16.886413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:43461 (local address 127.12.248.126:44581)
0925 10:25:16.887084 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:16.887091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:16.888274 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 10:25:16.891533 (+ 3259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:16.891541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:16.891629 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:16.891752 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:16.894266 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.895256 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.898700 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.899185 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.899336 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:16.904807 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:16.904817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:16.904823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:16.904877 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:16.908762 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.910042 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.910048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.910052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.910232 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.910985 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.910990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.910996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.911433 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:16.911593 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:16.912875 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0925 10:25:16.913129 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":416,"threads_started":1}
I20260925 10:25:16.916323 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:16.887002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.65:43461)
0925 10:25:16.888068 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:16.888121 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:16.891117 (+ 2996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:16.892093 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:16.892107 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:16.892909 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:16.894051 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:16.894081 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.895483 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.895492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:16.898474 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:16.898522 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.899442 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.899451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:16.899695 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:16.900471 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:16.900516 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:16.904512 (+ 3996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:16.908992 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:16.909005 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:16.909024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:16.909891 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:16.910371 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:16.910384 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:16.910391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:16.910816 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:16.911625 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:16.911639 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:16.912288 (+ 649us) client_negotiation.cc:769] Sending connection context
0925 10:25:16.912753 (+ 465us) client_negotiation.cc:240] Negotiation successful
0925 10:25:16.913483 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":452,"threads_started":1}
I20260925 10:25:16.932430 14477 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:41653
I20260925 10:25:16.932487 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:41653 every 8 connection(s)
I20260925 10:25:16.933981 14477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:16.935967 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14477
I20260925 10:25:16.936203 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:16.936492 14613 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44581
I20260925 10:25:16.936697 14613 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:16.937009 13281 external_mini_cluster.cc:1428] Setting key 6789c4c4f8d1ac4eb4cf0861f479c5b5
I20260925 10:25:16.937350 14613 heartbeater.cc:511] Master 127.12.248.126:44581 requested a full tablet report, sending...
I20260925 10:25:16.939975 14405 ts_manager.cc:194] Registered new tserver with Master: 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653)
I20260925 10:25:16.941113 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:16.925154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.65:43353)
0925 10:25:16.926089 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:16.926105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:16.926278 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:16.926716 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:16.926725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:16.927618 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:16.928243 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:16.928267 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.929758 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.929767 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:16.930929 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:16.930943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.932068 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.932072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:16.932213 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:16.932881 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:16.932898 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:16.933715 (+ 817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:16.937696 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:16.937702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:16.937706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:16.938321 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:16.938865 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:16.938870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:16.938874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:16.939041 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:16.939764 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:16.939772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:16.939973 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 10:25:16.940141 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:25:16.940517 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":663,"threads_started":1}
I20260925 10:25:16.942027 14405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:43461
I20260925 10:25:16.942251 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:16.924956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:43353 (local address 127.12.248.126:44581)
0925 10:25:16.925158 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:16.925167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:16.926204 (+ 1037us) server_negotiation.cc:407] Connection header received
0925 10:25:16.926419 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:16.926425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:16.926524 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:16.926629 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:16.928490 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.929532 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.931082 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:16.931614 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:16.931779 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:16.933906 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:16.933912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:16.933917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:16.933958 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:16.937554 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.938462 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.938469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.938473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.938686 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:16.939202 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:16.939207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:16.939211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:16.939604 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:16.939785 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:16.941420 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0925 10:25:16.941705 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:16.980748 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44581
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:17.313292 14617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:17.313661 14617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:17.313838 14617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:17.321585 14617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:17.321700 14617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:17.321837 14617 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:17.321964 14617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:17.332106 14617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:17.333406 14617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:17.334825 14617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:17.348122 14622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:17.348089 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:17.348027 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:17.348907 14617 server_base.cc:1034] running on GCE node
I20260925 10:25:17.349558 14617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:17.350426 14617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:17.351658 14617 hybrid_clock.cc:648] HybridClock initialized: now 1790331917351596 us; error 29 us; skew 500 ppm
I20260925 10:25:17.351976 14617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:17 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:17.362900 14617 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:17.364562 14617 webserver.cc:460] Webserver started at http://127.12.248.66:43195/ using document root <none> and password file <none>
I20260925 10:25:17.365271 14617 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:17.365360 14617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:17.365814 14617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:17.369062 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "1570a8edb8c242b3bf1654604507bb1b"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "79cf26b3421050ad7f22debf9dd705ba"
server_key_iv: "3af05378b24cde577fcae70000a6bd2b"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.369884 14617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "1570a8edb8c242b3bf1654604507bb1b"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "79cf26b3421050ad7f22debf9dd705ba"
server_key_iv: "3af05378b24cde577fcae70000a6bd2b"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.375306 14617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:25:17.378760 14632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:17.380215 14617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:17.380393 14617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "1570a8edb8c242b3bf1654604507bb1b"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "79cf26b3421050ad7f22debf9dd705ba"
server_key_iv: "3af05378b24cde577fcae70000a6bd2b"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.380602 14617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:17.394929 14617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:17.398939 14617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:17.399216 14617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:17.401939 14617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:17.402055 14617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:17.402232 14617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:17.402330 14617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:25:17 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:17.427227 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.402442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:56571 (local address 127.12.248.126:44581)
0925 10:25:17.402670 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:17.402680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:17.404145 (+ 1465us) server_negotiation.cc:407] Connection header received
0925 10:25:17.407230 (+ 3085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:17.407238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:17.407326 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:17.407455 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:17.410049 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.411055 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.414534 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.414987 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.415158 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.420192 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:17.420201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:17.420209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:17.420278 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:17.424044 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.424796 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.424802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.424805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.424931 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.425355 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.425360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.425364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.425723 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:17.425884 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:17.426650 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 10:25:17.426888 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:25:17.428280 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.403034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.66:56571)
0925 10:25:17.403941 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:17.404004 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:17.406832 (+ 2828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:17.407745 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:17.407761 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:17.408757 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:17.409799 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.409837 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.411267 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.411278 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.414381 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.414396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.415332 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.415342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.415755 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.416733 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:17.416777 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:17.419907 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:17.424191 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.424200 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.424224 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.424650 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.425040 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.425045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.425048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.425234 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.425874 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:17.425882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:17.426261 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 10:25:17.426530 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 10:25:17.427051 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":384,"threads_started":1}
I20260925 10:25:17.450238 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.435346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:51419 (local address 127.12.248.126:44581)
0925 10:25:17.435505 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:17.435509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:17.436500 (+ 991us) server_negotiation.cc:407] Connection header received
0925 10:25:17.436692 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:17.436695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:17.436780 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:17.436898 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:17.438616 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.439415 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.440710 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.441122 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.441275 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.444027 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:17.444036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:17.444042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:17.444086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:17.447153 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.448172 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.448176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.448178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.448297 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.448743 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.448746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.448748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.449169 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:17.449283 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:17.449838 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 10:25:17.450004 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:25:17.450932 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.435467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.66:51419)
0925 10:25:17.436366 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:17.436391 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:17.436569 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:17.436933 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:17.436942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:17.437683 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:17.438392 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.438413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.439555 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.439561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.440572 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.440583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.441323 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.441329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.441562 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.442349 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:17.442377 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:17.443789 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:17.447303 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.447312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.447319 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.448039 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.448411 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.448418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.448424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.448616 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.449291 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:17.449299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:17.449583 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 10:25:17.449772 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 10:25:17.450231 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":528,"threads_started":1}
I20260925 10:25:17.461661 14617 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:35975
I20260925 10:25:17.461695 14751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:35975 every 8 connection(s)
I20260925 10:25:17.462916 14617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:17.465059 14752 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44581
I20260925 10:25:17.465268 14752 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:17.465956 14752 heartbeater.cc:511] Master 127.12.248.126:44581 requested a full tablet report, sending...
I20260925 10:25:17.467207 14406 ts_manager.cc:194] Registered new tserver with Master: 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975)
I20260925 10:25:17.468158 14406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:56571
I20260925 10:25:17.469753 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14617
I20260925 10:25:17.469954 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:17.470489 13281 external_mini_cluster.cc:1428] Setting key 53e50c99683a7a875508f495b7fd2f90
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:17.509014 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44581
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:17.826277 14756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:17.826614 14756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:17.826799 14756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:17.834429 14756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:17.834581 14756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:17.834740 14756 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:17.834882 14756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:17.844673 14756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:44581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:17.846076 14756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:17.847515 14756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:17.860922 14761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:17.861122 14764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:17.861738 14762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:17.862609 14756 server_base.cc:1034] running on GCE node
I20260925 10:25:17.863286 14756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:17.864310 14756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:17.865566 14756 hybrid_clock.cc:648] HybridClock initialized: now 1790331917865488 us; error 59 us; skew 500 ppm
I20260925 10:25:17.865893 14756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:17 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:17.869745 14756 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:17.871270 14756 webserver.cc:460] Webserver started at http://127.12.248.67:45225/ using document root <none> and password file <none>
I20260925 10:25:17.871990 14756 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:17.872118 14756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:17.872478 14756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:17.875742 14756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "e34d36a006fd4d9bb0fccc0435103c9f"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "f344b38e0f3c76e8b313d6ffad1eb883"
server_key_iv: "e4c929f0ae48df5b6dff3a49b1b7f128"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.876540 14756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "e34d36a006fd4d9bb0fccc0435103c9f"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "f344b38e0f3c76e8b313d6ffad1eb883"
server_key_iv: "e4c929f0ae48df5b6dff3a49b1b7f128"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.882088 14756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:25:17.885919 14771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:17.887413 14756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:25:17.887595 14756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "e34d36a006fd4d9bb0fccc0435103c9f"
format_stamp: "Formatted at 2026-09-25 10:25:17 on dist-test-slave-f9rl"
server_key: "f344b38e0f3c76e8b313d6ffad1eb883"
server_key_iv: "e4c929f0ae48df5b6dff3a49b1b7f128"
server_key_version: "encryptionkey@0"
I20260925 10:25:17.887822 14756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:17.922957 14756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:17.927157 14756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:17.927455 14756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:17.930455 14756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:17.930573 14756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:17.930743 14756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:17.930833 14756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:17.945235 14613 heartbeater.cc:503] Master 127.12.248.126:44581 was elected leader, sending a full tablet report...
Sep 25 10:25:17 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331917, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:17.961527 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.931469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.67:35311)
0925 10:25:17.932396 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:17.932448 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:17.934405 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:17.935322 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:17.935333 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:17.936241 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:17.937234 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.937266 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.938975 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.938980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.941343 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.941365 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.942686 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.942691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.942886 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.943664 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:17.943687 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:17.951951 (+ 8264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:17.956455 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.956470 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.956499 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.957193 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.957702 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.957707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.957711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.958070 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.958743 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:17.958751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:17.959280 (+ 529us) client_negotiation.cc:769] Sending connection context
0925 10:25:17.959629 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 10:25:17.960208 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":398,"threads_started":1}
I20260925 10:25:17.962738 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.930850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:35311 (local address 127.12.248.126:44581)
0925 10:25:17.931077 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:17.931086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:17.932554 (+ 1468us) server_negotiation.cc:407] Connection header received
0925 10:25:17.934683 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:17.934689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:17.934796 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:17.934906 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:17.937518 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.938744 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.941553 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.942460 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.942638 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.952253 (+ 9615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:17.952260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:17.952267 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:17.952318 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:17.956266 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.957340 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.957353 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.957356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.957560 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.958238 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.958245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.958251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.958585 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:17.958747 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:17.962311 (+ 3564us) server_negotiation.cc:299] Negotiation successful
0925 10:25:17.962493 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:17.981336 14756 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:32821
I20260925 10:25:17.981370 14890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:32821 every 8 connection(s)
I20260925 10:25:17.982823 14756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:17.985088 14891 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44581
I20260925 10:25:17.985358 14891 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:17.985857 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.969574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.12.248.67:47739)
0925 10:25:17.970329 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:17.970352 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:17.970515 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:17.970896 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:17.970900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:17.971568 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:17.972108 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.972129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.973892 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.973900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.976093 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:17.976114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.977127 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.977134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:17.977344 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.978208 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:17.978239 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:17.979222 (+ 983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:17.982766 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.982771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.982775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.983297 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.983753 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:17.983760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:17.983767 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:17.983940 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:17.984735 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:17.984739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:17.984935 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:25:17.985102 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:25:17.985495 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":339,"threads_started":1}
I20260925 10:25:17.985920 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:17.969419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:47739 (local address 127.12.248.126:44581)
0925 10:25:17.969599 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:17.969603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:17.970430 (+ 827us) server_negotiation.cc:407] Connection header received
0925 10:25:17.970664 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:17.970669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:17.970759 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:17.970865 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:17.972289 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.973651 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.976284 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:17.976916 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:17.977078 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:17.979530 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:17.979535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:17.979538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:17.979571 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:17.982620 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.983444 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.983449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.983458 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.983579 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:17.984084 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:17.984091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:17.984097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:17.984588 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:17.984745 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:17.985278 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 10:25:17.985554 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:25:17.986285 14891 heartbeater.cc:511] Master 127.12.248.126:44581 requested a full tablet report, sending...
I20260925 10:25:17.986495 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 14756
I20260925 10:25:17.986685 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:17.987234 13281 external_mini_cluster.cc:1428] Setting key d96e99a425165cc29939fcd5873492a9
I20260925 10:25:17.989468 14406 ts_manager.cc:194] Registered new tserver with Master: e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821)
I20260925 10:25:17.990514 14406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:35311
I20260925 10:25:17.990556 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:25:18 dist-test-slave-f9rl krb5kdc[14361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:25:18 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:18.035120 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.021711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.12.248.126:44581)
0925 10:25:18.021888 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.021893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.022187 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:25:18.022355 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.022359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.022426 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.022523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:18.023878 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.024614 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.025935 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.026357 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.026451 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.030181 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:18.030185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:18.030188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:18.030226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:18.032907 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:18.033589 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:18.033593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:18.033595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:18.033705 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:18.034021 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:18.034024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:18.034026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:18.034288 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:18.034460 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.034687 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.034849 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:25:18.039201 14406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:18.046111 14406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:18.077880 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.059807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.12.248.66:35975)
0925 10:25:18.069221 (+ 9414us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.069230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.069955 (+ 725us) server_negotiation.cc:407] Connection header received
0925 10:25:18.070073 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.070084 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.070374 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.071117 (+ 743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.071318 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.072539 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.075296 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.076431 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.076555 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.076774 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.077366 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.077576 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9246,"thread_start_us":9631,"threads_started":1}
I20260925 10:25:18.077976 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.058504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:35975 (local address 127.0.0.1:35726)
0925 10:25:18.069317 (+ 10813us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.069380 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.069776 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.070614 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.070622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.070655 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.071141 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.071159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.072680 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.072688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.075145 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.075156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.076909 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.076915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.077099 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.077152 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.077298 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.077524 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10521,"mutex_wait_us":22,"thread_start_us":11546,"threads_started":1}
I20260925 10:25:18.078419 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.058640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32821 (local address 127.0.0.1:47254)
0925 10:25:18.070364 (+ 11724us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.070392 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.070527 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.071347 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.071353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.071379 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.071765 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.071776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.073624 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.073631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.075694 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.075723 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.077574 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.077580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.077755 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.077786 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.077866 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.078009 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11590,"mutex_wait_us":4,"thread_start_us":10603,"threads_started":1}
I20260925 10:25:18.079341 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.060743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:41653 (local address 127.0.0.1:59220)
0925 10:25:18.070750 (+ 10007us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.070768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.070922 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.072259 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.072264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.072294 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.072755 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.072768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.074681 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.074687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.076793 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.076807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.078601 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.078607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.078776 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.078818 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.078915 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.079033 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9892,"thread_start_us":9751,"threads_started":1}
I20260925 10:25:18.079420 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.059238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.12.248.67:32821)
0925 10:25:18.060151 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.060165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.070674 (+ 10509us) server_negotiation.cc:407] Connection header received
0925 10:25:18.070816 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.070825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.071044 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.071258 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.071991 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.073497 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.075881 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.077330 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.078616 (+ 1286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.078856 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.078926 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.079073 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":726,"threads_started":1}
I20260925 10:25:18.079682 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.061417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.12.248.65:41653)
0925 10:25:18.068441 (+ 7024us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.068455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.070881 (+ 2426us) server_negotiation.cc:407] Connection header received
0925 10:25:18.071619 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.071628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.071863 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.072036 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.072996 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.074544 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.077014 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.078376 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.078524 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.078729 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.079236 (+ 507us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.079404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6813,"thread_start_us":6427,"threads_started":1}
I20260925 10:25:18.082106 14682 tablet_service.cc:1467] Processing CreateTablet for tablet a21f46b484484addb9d49e6021bb66f3 (DEFAULT_TABLE table=test-table [id=7c06fbb042654dfab7c42f945918368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.082782 14821 tablet_service.cc:1467] Processing CreateTablet for tablet a21f46b484484addb9d49e6021bb66f3 (DEFAULT_TABLE table=test-table [id=7c06fbb042654dfab7c42f945918368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.082909 14543 tablet_service.cc:1467] Processing CreateTablet for tablet a21f46b484484addb9d49e6021bb66f3 (DEFAULT_TABLE table=test-table [id=7c06fbb042654dfab7c42f945918368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.083632 14682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f46b484484addb9d49e6021bb66f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.084106 14821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f46b484484addb9d49e6021bb66f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.084105 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21f46b484484addb9d49e6021bb66f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.095661 14910 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Bootstrap starting.
I20260925 10:25:18.099203 14910 tablet_bootstrap.cc:654] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.100236 14910 log.cc:824] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:18.102218 14912 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Bootstrap starting.
I20260925 10:25:18.103514 14910 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: No bootstrap required, opened a new log
I20260925 10:25:18.103809 14910 ts_tablet_manager.cc:1397] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:25:18.105556 14911 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Bootstrap starting.
I20260925 10:25:18.105871 14912 tablet_bootstrap.cc:654] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.107149 14912 log.cc:824] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:18.108758 14911 tablet_bootstrap.cc:654] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.110085 14911 log.cc:824] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:18.110380 14912 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: No bootstrap required, opened a new log
I20260925 10:25:18.110682 14912 ts_tablet_manager.cc:1397] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 10:25:18.113160 14911 tablet_bootstrap.cc:492] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: No bootstrap required, opened a new log
I20260925 10:25:18.113479 14911 ts_tablet_manager.cc:1397] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260925 10:25:18.113188 14910 raft_consensus.cc:348] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.113765 14910 raft_consensus.cc:374] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.113873 14910 raft_consensus.cc:729] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1570a8edb8c242b3bf1654604507bb1b, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.114526 14910 consensus_queue.cc:260] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.116024 14752 heartbeater.cc:503] Master 127.12.248.126:44581 was elected leader, sending a full tablet report...
I20260925 10:25:18.116771 14910 ts_tablet_manager.cc:1428] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 10:25:18.121690 14912 raft_consensus.cc:348] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.122201 14912 raft_consensus.cc:374] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.122319 14912 raft_consensus.cc:729] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34d36a006fd4d9bb0fccc0435103c9f, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.123076 14912 consensus_queue.cc:260] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.124229 14891 heartbeater.cc:503] Master 127.12.248.126:44581 was elected leader, sending a full tablet report...
I20260925 10:25:18.124569 14911 raft_consensus.cc:348] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.125037 14912 ts_tablet_manager.cc:1428] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 10:25:18.125067 14911 raft_consensus.cc:374] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.125212 14911 raft_consensus.cc:729] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929c64ff70954b24b644b5b0c1f15f9e, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.125885 14911 consensus_queue.cc:260] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.127491 14911 ts_tablet_manager.cc:1428] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
W20260925 10:25:18.186522 14614 tablet.cc:2369] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:25:18.215056 14753 tablet.cc:2369] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:25:18.234656 14892 tablet.cc:2369] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:18.332345 14916 raft_consensus.cc:482] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:18.332710 14916 raft_consensus.cc:504] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.334568 14916 leader_election.cc:288] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821), 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653)
I20260925 10:25:18.344226 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.335506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:57999 (local address 127.12.248.65:41653)
0925 10:25:18.335717 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.335723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.336019 (+ 296us) server_negotiation.cc:407] Connection header received
0925 10:25:18.336395 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.336405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.336511 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.336580 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.337483 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.338630 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.341798 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.343066 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.343234 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.343311 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.343816 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.343947 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:25:18.344532 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.334940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:41653 (local address 127.12.248.66:57999)
0925 10:25:18.335889 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.335910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.336174 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.336811 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.336816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.336849 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.337299 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.337309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.338790 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.338795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.341600 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.341619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.343353 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.343358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.343590 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.343673 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.343772 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.343936 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":205,"thread_start_us":663,"threads_started":1}
I20260925 10:25:18.344627 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.335102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:44603 (local address 127.12.248.67:32821)
0925 10:25:18.335358 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.335368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.336337 (+ 969us) server_negotiation.cc:407] Connection header received
0925 10:25:18.336602 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.336607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.336677 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.336733 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.337653 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.338697 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.341785 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.343059 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.343251 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.343317 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.343816 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.343947 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:25:18.344558 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.334962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32821 (local address 127.12.248.66:44603)
0925 10:25:18.336227 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.336246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.336446 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.336937 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.336942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.336983 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.337477 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.337489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.338817 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.338824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.341600 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.341619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.343340 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.343347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.343585 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.343669 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.343772 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.343936 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":264,"thread_start_us":1730,"threads_started":1}
I20260925 10:25:18.345824 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f46b484484addb9d49e6021bb66f3" candidate_uuid: "1570a8edb8c242b3bf1654604507bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929c64ff70954b24b644b5b0c1f15f9e" is_pre_election: true
I20260925 10:25:18.345824 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f46b484484addb9d49e6021bb66f3" candidate_uuid: "1570a8edb8c242b3bf1654604507bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" is_pre_election: true
I20260925 10:25:18.346351 14563 raft_consensus.cc:2454] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1570a8edb8c242b3bf1654604507bb1b in term 0.
I20260925 10:25:18.346351 14841 raft_consensus.cc:2454] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1570a8edb8c242b3bf1654604507bb1b in term 0.
I20260925 10:25:18.347038 14634 leader_election.cc:302] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1570a8edb8c242b3bf1654604507bb1b, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.347468 14916 raft_consensus.cc:2781] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:18.347612 14916 raft_consensus.cc:482] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:18.347738 14916 raft_consensus.cc:3037] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.350024 14916 raft_consensus.cc:504] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.350656 14916 leader_election.cc:288] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [CANDIDATE]: Term 1 election: Requested vote from peers e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821), 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653)
I20260925 10:25:18.351130 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f46b484484addb9d49e6021bb66f3" candidate_uuid: "1570a8edb8c242b3bf1654604507bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929c64ff70954b24b644b5b0c1f15f9e"
I20260925 10:25:18.351130 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21f46b484484addb9d49e6021bb66f3" candidate_uuid: "1570a8edb8c242b3bf1654604507bb1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34d36a006fd4d9bb0fccc0435103c9f"
I20260925 10:25:18.351339 14841 raft_consensus.cc:3037] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.351339 14563 raft_consensus.cc:3037] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.354588 14563 raft_consensus.cc:2454] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1570a8edb8c242b3bf1654604507bb1b in term 1.
I20260925 10:25:18.354588 14841 raft_consensus.cc:2454] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1570a8edb8c242b3bf1654604507bb1b in term 1.
I20260925 10:25:18.355023 14634 leader_election.cc:302] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1570a8edb8c242b3bf1654604507bb1b, 929c64ff70954b24b644b5b0c1f15f9e, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.355392 14916 raft_consensus.cc:2781] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:18.355835 14916 raft_consensus.cc:686] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 1 LEADER]: Becoming Leader. State: Replica: 1570a8edb8c242b3bf1654604507bb1b, State: Running, Role: LEADER
I20260925 10:25:18.356374 14916 consensus_queue.cc:237] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } }
I20260925 10:25:18.360757 14405 catalog_manager.cc:5495] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66). New cstate: current_term: 1 leader_uuid: "1570a8edb8c242b3bf1654604507bb1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } health_report { overall_health: UNKNOWN } } }
I20260925 10:25:18.443264 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.438085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.12.248.66:35975)
0925 10:25:18.438588 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.438593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.438612 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:25:18.438691 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.438695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.438770 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.438887 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:18.439188 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.439898 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.441144 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.441596 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.441687 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.442119 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:18.442674 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:18.442782 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.442894 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.442957 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260925 10:25:18.451910 14841 raft_consensus.cc:1261] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Refusing update from remote peer 1570a8edb8c242b3bf1654604507bb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:18.452857 14921 consensus_queue.cc:1035] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:18.453778 14563 raft_consensus.cc:1261] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Refusing update from remote peer 1570a8edb8c242b3bf1654604507bb1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:18.454618 14921 consensus_queue.cc:1035] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:18.466876 14925 mvcc.cc:205] Tried to move back new op lower bound from 7333199537967779840 to 7333199537591164928. Current Snapshot: MvccSnapshot[applied={T|T < 7333199537967779840}]
I20260925 10:25:18.481145 14926 mvcc.cc:205] Tried to move back new op lower bound from 7333199537967779840 to 7333199537591164928. Current Snapshot: MvccSnapshot[applied={T|T < 7333199537967779840}]
I20260925 10:25:18.502912 14405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:18.503342 14405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:18.507766 14405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Sending DeleteTablet for 3 replicas of tablet a21f46b484484addb9d49e6021bb66f3
I20260925 10:25:18.509111 14682 tablet_service.cc:1514] Processing DeleteTablet for tablet a21f46b484484addb9d49e6021bb66f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:18 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:35726
I20260925 10:25:18.509312 14821 tablet_service.cc:1514] Processing DeleteTablet for tablet a21f46b484484addb9d49e6021bb66f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:18 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:47254
I20260925 10:25:18.509317 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet a21f46b484484addb9d49e6021bb66f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:18 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59220
I20260925 10:25:18.510998 14405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:18.512125 14405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:18.520362 14936 tablet_replica.cc:330] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: stopping tablet replica
I20260925 10:25:18.522119 14936 raft_consensus.cc:2229] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:18.523336 14820 tablet_service.cc:1467] Processing CreateTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf (DEFAULT_TABLE table=test-table [id=498687aed4d74f32b5decfadbc17f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.523845 14820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f423b8d88fc42aeaf5637998a7b40bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.527025 14681 tablet_service.cc:1467] Processing CreateTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf (DEFAULT_TABLE table=test-table [id=498687aed4d74f32b5decfadbc17f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.527818 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f423b8d88fc42aeaf5637998a7b40bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.528249 14935 tablet_replica.cc:330] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: stopping tablet replica
I20260925 10:25:18.528785 14935 raft_consensus.cc:2229] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:18.529296 14935 raft_consensus.cc:2258] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:18.529971 14934 tablet_replica.cc:330] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: stopping tablet replica
I20260925 10:25:18.530432 14934 raft_consensus.cc:2229] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:18.530650 14936 raft_consensus.cc:2258] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:18.531306 14912 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Bootstrap starting.
I20260925 10:25:18.524158 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf (DEFAULT_TABLE table=test-table [id=498687aed4d74f32b5decfadbc17f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:18.531306 14934 raft_consensus.cc:2258] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:18.532291 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f423b8d88fc42aeaf5637998a7b40bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.533047 14912 tablet_bootstrap.cc:654] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.533643 14934 ts_tablet_manager.cc:1905] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:18.536870 14910 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Bootstrap starting.
I20260925 10:25:18.538663 14910 tablet_bootstrap.cc:654] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.538923 14911 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Bootstrap starting.
I20260925 10:25:18.540493 14911 tablet_bootstrap.cc:654] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.543675 14935 ts_tablet_manager.cc:1905] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:18.547583 14934 ts_tablet_manager.cc:1918] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:18.547848 14934 log.cc:1196] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/a21f46b484484addb9d49e6021bb66f3
I20260925 10:25:18.548636 14934 ts_tablet_manager.cc:1939] T a21f46b484484addb9d49e6021bb66f3 P 1570a8edb8c242b3bf1654604507bb1b: Deleting consensus metadata
I20260925 10:25:18.550947 14392 catalog_manager.cc:4841] TS 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975): tablet a21f46b484484addb9d49e6021bb66f3 (table test-table [id=7c06fbb042654dfab7c42f945918368a]) successfully deleted
I20260925 10:25:18.551108 14936 ts_tablet_manager.cc:1905] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:18.559932 14935 ts_tablet_manager.cc:1918] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:18.560228 14935 log.cc:1196] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/a21f46b484484addb9d49e6021bb66f3
I20260925 10:25:18.560946 14935 ts_tablet_manager.cc:1939] T a21f46b484484addb9d49e6021bb66f3 P e34d36a006fd4d9bb0fccc0435103c9f: Deleting consensus metadata
I20260925 10:25:18.563472 14391 catalog_manager.cc:4841] TS e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821): tablet a21f46b484484addb9d49e6021bb66f3 (table test-table [id=7c06fbb042654dfab7c42f945918368a]) successfully deleted
I20260925 10:25:18.563814 14911 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: No bootstrap required, opened a new log
I20260925 10:25:18.564042 14911 ts_tablet_manager.cc:1397] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.002s
I20260925 10:25:18.565071 14911 raft_consensus.cc:348] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.565402 14911 raft_consensus.cc:374] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.565562 14911 raft_consensus.cc:729] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929c64ff70954b24b644b5b0c1f15f9e, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.566447 14911 consensus_queue.cc:260] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.568707 14910 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: No bootstrap required, opened a new log
I20260925 10:25:18.568920 14910 ts_tablet_manager.cc:1397] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.012s
I20260925 10:25:18.569852 14910 raft_consensus.cc:348] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.570045 14911 ts_tablet_manager.cc:1428] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 10:25:18.570164 14910 raft_consensus.cc:374] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.570331 14910 raft_consensus.cc:729] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1570a8edb8c242b3bf1654604507bb1b, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.570544 14936 ts_tablet_manager.cc:1918] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:18.570667 14910 consensus_queue.cc:260] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.571727 14912 tablet_bootstrap.cc:492] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: No bootstrap required, opened a new log
I20260925 10:25:18.571893 14912 ts_tablet_manager.cc:1397] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.004s
I20260925 10:25:18.572717 14910 ts_tablet_manager.cc:1428] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:25:18.572762 14912 raft_consensus.cc:348] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.573024 14912 raft_consensus.cc:374] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.573154 14912 raft_consensus.cc:729] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34d36a006fd4d9bb0fccc0435103c9f, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.573467 14912 consensus_queue.cc:260] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.575237 14912 ts_tablet_manager.cc:1428] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:25:18.575523 14936 log.cc:1196] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/a21f46b484484addb9d49e6021bb66f3
I20260925 10:25:18.576290 14936 ts_tablet_manager.cc:1939] T a21f46b484484addb9d49e6021bb66f3 P 929c64ff70954b24b644b5b0c1f15f9e: Deleting consensus metadata
I20260925 10:25:18.578294 14393 catalog_manager.cc:4841] TS 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653): tablet a21f46b484484addb9d49e6021bb66f3 (table test-table [id=7c06fbb042654dfab7c42f945918368a]) successfully deleted
I20260925 10:25:18.744031 14918 raft_consensus.cc:482] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:18.744307 14918 raft_consensus.cc:504] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.746304 14918 leader_election.cc:288] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975), e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821)
I20260925 10:25:18.755993 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.747048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:52489 (local address 127.12.248.66:35975)
0925 10:25:18.747247 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.747256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.748632 (+ 1376us) server_negotiation.cc:407] Connection header received
0925 10:25:18.748880 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.748886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.748969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.749085 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.749880 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.751061 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.753079 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.754446 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.754667 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.754761 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.755415 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.755572 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:25:18.755975 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.746714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:35975 (local address 127.12.248.65:52489)
0925 10:25:18.748540 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.748560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.748725 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.749100 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.749105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.749147 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.749630 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.749644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.751194 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.751204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.752889 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.752903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.754933 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.754941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.755157 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.755203 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.755385 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.755534 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":9,"thread_start_us":1111,"threads_started":1}
I20260925 10:25:18.756352 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.747370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:38689 (local address 127.12.248.67:32821)
0925 10:25:18.747627 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.747637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.748325 (+ 688us) server_negotiation.cc:407] Connection header received
0925 10:25:18.748677 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.748683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.748780 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.748890 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.749759 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.750983 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.753707 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.754996 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.755163 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.755244 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.755791 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.755989 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:25:18.756963 14941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.746682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32821 (local address 127.12.248.65:38689)
0925 10:25:18.748188 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.748212 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.748469 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.748909 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.748913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.748935 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.749556 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.749571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.751148 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.751154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.753508 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.753525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.755259 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.755276 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.755515 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.755564 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.756394 (+ 830us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.756535 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":49,"thread_start_us":1524,"threads_started":1}
I20260925 10:25:18.757674 14707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f423b8d88fc42aeaf5637998a7b40bf" candidate_uuid: "929c64ff70954b24b644b5b0c1f15f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570a8edb8c242b3bf1654604507bb1b" is_pre_election: true
I20260925 10:25:18.757848 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f423b8d88fc42aeaf5637998a7b40bf" candidate_uuid: "929c64ff70954b24b644b5b0c1f15f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" is_pre_election: true
I20260925 10:25:18.758117 14841 raft_consensus.cc:2454] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 929c64ff70954b24b644b5b0c1f15f9e in term 0.
I20260925 10:25:18.758203 14707 raft_consensus.cc:2454] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 929c64ff70954b24b644b5b0c1f15f9e in term 0.
I20260925 10:25:18.758666 14494 leader_election.cc:302] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 929c64ff70954b24b644b5b0c1f15f9e, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.759184 14918 raft_consensus.cc:2781] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:18.759387 14918 raft_consensus.cc:482] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:18.759543 14918 raft_consensus.cc:3037] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.763020 14918 raft_consensus.cc:504] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.763995 14918 leader_election.cc:288] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [CANDIDATE]: Term 1 election: Requested vote from peers 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975), e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821)
I20260925 10:25:18.764338 14707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f423b8d88fc42aeaf5637998a7b40bf" candidate_uuid: "929c64ff70954b24b644b5b0c1f15f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570a8edb8c242b3bf1654604507bb1b"
I20260925 10:25:18.764565 14707 raft_consensus.cc:3037] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.764664 14841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f423b8d88fc42aeaf5637998a7b40bf" candidate_uuid: "929c64ff70954b24b644b5b0c1f15f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e34d36a006fd4d9bb0fccc0435103c9f"
I20260925 10:25:18.764962 14841 raft_consensus.cc:3037] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.767885 14841 raft_consensus.cc:2454] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 929c64ff70954b24b644b5b0c1f15f9e in term 1.
I20260925 10:25:18.768214 14707 raft_consensus.cc:2454] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 929c64ff70954b24b644b5b0c1f15f9e in term 1.
I20260925 10:25:18.768349 14494 leader_election.cc:302] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 929c64ff70954b24b644b5b0c1f15f9e, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.768736 14918 raft_consensus.cc:2781] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:18.769210 14918 raft_consensus.cc:686] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 1 LEADER]: Becoming Leader. State: Replica: 929c64ff70954b24b644b5b0c1f15f9e, State: Running, Role: LEADER
I20260925 10:25:18.769955 14918 consensus_queue.cc:237] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.773770 14405 catalog_manager.cc:5495] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65). New cstate: current_term: 1 leader_uuid: "929c64ff70954b24b644b5b0c1f15f9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:25:18 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331916, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:18.876370 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.860069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44581 (local address 127.0.0.1:40186)
0925 10:25:18.860767 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.860790 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.860965 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.861338 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.861344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.862048 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:18.862266 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.862275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.863865 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.863872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.865547 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.865566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.866682 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.866688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.866878 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.867619 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:18.867702 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:18.870508 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:18.873716 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:18.873733 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:18.873742 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:18.874216 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:18.874583 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:18.874589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:18.874593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:18.874756 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:18.875251 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:18.875260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:18.875423 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.875577 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.875866 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":344,"threads_started":1}
I20260925 10:25:18.876382 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.860117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.12.248.126:44581)
0925 10:25:18.861062 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.861070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.861099 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:25:18.861140 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.861143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.861192 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.861311 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:18.862445 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.863637 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.865821 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.866497 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.866601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.870706 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:18.870710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:18.870714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:18.870757 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:18.873593 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:18.874345 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:18.874351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:18.874357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:18.874479 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:18.874866 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:18.874870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:18.874874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:18.875121 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:18.875267 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.875665 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.876011 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":814,"thread_start_us":667,"threads_started":1}
I20260925 10:25:18.882283 14405 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:40186:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:25:18.883502 14405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:18.890847 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31dfb69aaf459c93f56e4becdb8e9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cfaa2e4f87e49cca2f5699103fb0d1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:18.890981 14681 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31dfb69aaf459c93f56e4becdb8e9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cfaa2e4f87e49cca2f5699103fb0d1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:18.891446 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31dfb69aaf459c93f56e4becdb8e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.891460 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31dfb69aaf459c93f56e4becdb8e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.891649 14820 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31dfb69aaf459c93f56e4becdb8e9e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9cfaa2e4f87e49cca2f5699103fb0d1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:18.892210 14820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31dfb69aaf459c93f56e4becdb8e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:18.899449 14911 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: Bootstrap starting.
I20260925 10:25:18.902015 14911 tablet_bootstrap.cc:654] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.904803 14910 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: Bootstrap starting.
I20260925 10:25:18.905257 14912 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: Bootstrap starting.
I20260925 10:25:18.907351 14912 tablet_bootstrap.cc:654] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.907516 14910 tablet_bootstrap.cc:654] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:18.911430 14911 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: No bootstrap required, opened a new log
I20260925 10:25:18.911746 14911 ts_tablet_manager.cc:1397] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260925 10:25:18.912822 14911 raft_consensus.cc:348] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.913251 14911 raft_consensus.cc:374] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.913399 14911 raft_consensus.cc:729] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929c64ff70954b24b644b5b0c1f15f9e, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.913843 14911 consensus_queue.cc:260] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.914474 14918 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:18.914691 14911 ts_tablet_manager.cc:1428] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:25:18.914829 14918 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:18.916507 14910 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: No bootstrap required, opened a new log
I20260925 10:25:18.916795 14910 ts_tablet_manager.cc:1397] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260925 10:25:18.916960 14912 tablet_bootstrap.cc:492] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: No bootstrap required, opened a new log
I20260925 10:25:18.917188 14912 ts_tablet_manager.cc:1397] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20260925 10:25:18.917964 14910 raft_consensus.cc:348] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.918367 14910 raft_consensus.cc:374] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.918515 14910 raft_consensus.cc:729] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1570a8edb8c242b3bf1654604507bb1b, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.918385 14912 raft_consensus.cc:348] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.918656 14912 raft_consensus.cc:374] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:18.918764 14912 raft_consensus.cc:729] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34d36a006fd4d9bb0fccc0435103c9f, State: Initialized, Role: FOLLOWER
I20260925 10:25:18.919029 14910 consensus_queue.cc:260] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.919128 14912 consensus_queue.cc:260] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.919557 14917 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:18.919617 14916 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:18.919973 14917 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:18.920089 14916 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:18.920778 14910 ts_tablet_manager.cc:1428] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:25:18.920974 14912 ts_tablet_manager.cc:1428] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 10:25:18.942525 14917 raft_consensus.cc:482] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:18.942763 14917 raft_consensus.cc:504] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.944625 14917 leader_election.cc:288] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653), 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975)
I20260925 10:25:18.966979 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.945281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:58193 (local address 127.12.248.66:35975)
0925 10:25:18.945549 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.945556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.959198 (+ 13642us) server_negotiation.cc:407] Connection header received
0925 10:25:18.959499 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.959506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.959597 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.959666 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.960537 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.961720 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.964259 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.965604 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.965801 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.965907 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.966479 (+ 572us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.966634 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:25:18.968453 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.944826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:35975 (local address 127.12.248.67:58193)
0925 10:25:18.959099 (+ 14273us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.959116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.959327 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.959886 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.959891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.959918 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.960330 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.960342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.961908 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.961913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.964078 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.964090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.966069 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.966076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.966282 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.966317 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.967765 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.967958 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14142,"mutex_wait_us":5,"spinlock_wait_cycles":5760,"thread_start_us":13978,"threads_started":1}
I20260925 10:25:18.968945 14954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.944757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:41653 (local address 127.12.248.67:44561)
0925 10:25:18.958345 (+ 13588us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:18.958363 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:18.958532 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:18.959101 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:18.959107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:18.959146 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:18.959536 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.959548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.961168 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.961174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.963729 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:18.963743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.968085 (+ 4342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.968091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:18.968266 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.968349 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 10:25:18.968462 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:25:18.968609 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13459,"spinlock_wait_cycles":34048,"thread_start_us":13308,"threads_started":1}
I20260925 10:25:18.969383 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:18.945110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:44561 (local address 127.12.248.65:41653)
0925 10:25:18.957748 (+ 12638us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:18.957758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:18.958455 (+ 697us) server_negotiation.cc:407] Connection header received
0925 10:25:18.958673 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:18.958680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:18.958767 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:18.958868 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:18.959711 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.961004 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.963923 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:18.967610 (+ 3687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:18.967748 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:18.967816 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:18.968869 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0925 10:25:18.969084 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12427,"thread_start_us":12641,"threads_started":1}
I20260925 10:25:18.969313 14707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31dfb69aaf459c93f56e4becdb8e9e" candidate_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570a8edb8c242b3bf1654604507bb1b" is_pre_election: true
I20260925 10:25:18.969648 14707 raft_consensus.cc:2454] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e34d36a006fd4d9bb0fccc0435103c9f in term 0.
I20260925 10:25:18.970002 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31dfb69aaf459c93f56e4becdb8e9e" candidate_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929c64ff70954b24b644b5b0c1f15f9e" is_pre_election: true
I20260925 10:25:18.970242 14563 raft_consensus.cc:2454] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e34d36a006fd4d9bb0fccc0435103c9f in term 0.
I20260925 10:25:18.970350 14774 leader_election.cc:302] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1570a8edb8c242b3bf1654604507bb1b, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.971379 14917 raft_consensus.cc:2781] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:18.971527 14917 raft_consensus.cc:482] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:18.971659 14917 raft_consensus.cc:3037] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.975397 14917 raft_consensus.cc:504] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:18.976377 14917 leader_election.cc:288] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [CANDIDATE]: Term 1 election: Requested vote from peers 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653), 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975)
I20260925 10:25:18.976619 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31dfb69aaf459c93f56e4becdb8e9e" candidate_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929c64ff70954b24b644b5b0c1f15f9e"
I20260925 10:25:18.976662 14707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31dfb69aaf459c93f56e4becdb8e9e" candidate_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570a8edb8c242b3bf1654604507bb1b"
I20260925 10:25:18.976843 14563 raft_consensus.cc:3037] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.976897 14707 raft_consensus.cc:3037] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:18.979859 14563 raft_consensus.cc:2454] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e34d36a006fd4d9bb0fccc0435103c9f in term 1.
I20260925 10:25:18.980028 14707 raft_consensus.cc:2454] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e34d36a006fd4d9bb0fccc0435103c9f in term 1.
I20260925 10:25:18.980505 14774 leader_election.cc:302] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1570a8edb8c242b3bf1654604507bb1b, e34d36a006fd4d9bb0fccc0435103c9f; no voters:
I20260925 10:25:18.981207 14917 raft_consensus.cc:2781] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:18.997727 14917 raft_consensus.cc:686] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [term 1 LEADER]: Becoming Leader. State: Replica: e34d36a006fd4d9bb0fccc0435103c9f, State: Running, Role: LEADER
I20260925 10:25:18.998327 14917 consensus_queue.cc:237] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } }
I20260925 10:25:19.002964 14957 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: TxnStatusTablet state changed. Reason: New leader e34d36a006fd4d9bb0fccc0435103c9f. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.003384 14957 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:19.004376 14406 catalog_manager.cc:5495] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67). New cstate: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } health_report { overall_health: HEALTHY } } }
I20260925 10:25:19.010432 14959 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:25:19.010921 14959 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:25:19.044486 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.036479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.12.248.67:32821)
0925 10:25:19.036648 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.036653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.036670 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:25:19.036744 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.036747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.036840 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.037004 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:19.037632 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.038805 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.041503 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.042237 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.042367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.043039 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.043766 (+ 727us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.043909 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.044071 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.044164 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:25:19.044615 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.035864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32821 (local address 127.0.0.1:47264)
0925 10:25:19.036200 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:19.036225 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:19.036380 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:19.037026 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:19.037031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:19.037043 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:19.037463 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.037480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.038964 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.038970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.041303 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.041321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.042422 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.042429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.042685 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.042788 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.043899 (+ 1111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.043927 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:25:19.044035 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:25:19.044232 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:25:19.050529 14563 raft_consensus.cc:1261] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Refusing update from remote peer e34d36a006fd4d9bb0fccc0435103c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:19.050726 14707 raft_consensus.cc:1261] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Refusing update from remote peer e34d36a006fd4d9bb0fccc0435103c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:19.051409 14957 consensus_queue.cc:1035] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:19.051753 14917 consensus_queue.cc:1035] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:19.063694 14918 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: TxnStatusTablet state changed. Reason: New leader e34d36a006fd4d9bb0fccc0435103c9f. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.064368 14918 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:19.065815 14916 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: TxnStatusTablet state changed. Reason: New leader e34d36a006fd4d9bb0fccc0435103c9f. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.066403 14916 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:19.068885 14964 mvcc.cc:205] Tried to move back new op lower bound from 7333199540422778880 to 7333199540221800448. Current Snapshot: MvccSnapshot[applied={T|T < 7333199540422778880}]
I20260925 10:25:19.069911 14916 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.070233 14916 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 1570a8edb8c242b3bf1654604507bb1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:19.072113 14957 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.072386 14957 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:19.072633 14957 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.072896 14957 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P e34d36a006fd4d9bb0fccc0435103c9f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:19.075404 14918 tablet_replica.cc:439] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929c64ff70954b24b644b5b0c1f15f9e" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 41653 } } peers { permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 } } peers { permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 } } }
I20260925 10:25:19.075827 14918 tablet_replica.cc:442] T 2d31dfb69aaf459c93f56e4becdb8e9e P 929c64ff70954b24b644b5b0c1f15f9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:19.087867 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.079230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.12.248.65:41653)
0925 10:25:19.079409 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.079418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.080005 (+ 587us) server_negotiation.cc:407] Connection header received
0925 10:25:19.080188 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.080194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.080338 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.080522 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:19.081116 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.082493 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.084494 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.085503 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.085700 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.086200 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.087240 (+ 1040us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.087395 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.087451 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.087543 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:25:19.110450 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.103774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32821 (local address 127.12.248.65:43999)
0925 10:25:19.104372 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:19.104387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:19.104521 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:19.105010 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:19.105013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:19.105023 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:19.105354 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.105361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.107004 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.107011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.108191 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.108203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.109301 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.109305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.109416 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.109536 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.109996 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.110005 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:25:19.110064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:25:19.110160 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260925 10:25:19.110738 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.103753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:43999 (local address 127.12.248.67:32821)
0925 10:25:19.103929 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.103934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.104525 (+ 591us) server_negotiation.cc:407] Connection header received
0925 10:25:19.104643 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.104649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.104834 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.105006 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:19.105576 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.106774 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.108340 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.109130 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.109282 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.109741 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.109892 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.109999 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.110098 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.110190 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:25:19.122802 14707 raft_consensus.cc:1261] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Refusing update from remote peer 929c64ff70954b24b644b5b0c1f15f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:19.122823 14841 raft_consensus.cc:1261] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Refusing update from remote peer 929c64ff70954b24b644b5b0c1f15f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:19.123497 14918 consensus_queue.cc:1035] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1570a8edb8c242b3bf1654604507bb1b" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:19.123826 14942 consensus_queue.cc:1035] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e34d36a006fd4d9bb0fccc0435103c9f" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:19.176362 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.166886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37919 (local address 127.12.248.65:41653)
0925 10:25:19.167090 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.167111 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.170299 (+ 3188us) server_negotiation.cc:407] Connection header received
0925 10:25:19.170505 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.170513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.170617 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.170715 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:19.171342 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.172055 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.174288 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.174944 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.175068 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.175607 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.175727 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.175822 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.175962 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.176039 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:25:19.176533 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.166975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:41653 (local address 127.12.248.67:37919)
0925 10:25:19.170183 (+ 3208us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:19.170203 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:19.170353 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:19.170748 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:19.170757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:19.170772 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:19.171159 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.171177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.172177 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.172183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.174138 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:19.174153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.175232 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.175239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:19.175374 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.175428 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:19.175827 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:19.175840 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:25:19.175940 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:25:19.176083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2995,"thread_start_us":2790,"threads_started":1}
I20260925 10:25:19.222929 14406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:19.223199 14406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:19.226939 14406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4eeda86f85364cec8e98ebc9d89ec0fe: Sending DeleteTablet for 3 replicas of tablet 1f423b8d88fc42aeaf5637998a7b40bf
I20260925 10:25:19.228056 14681 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:19 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:35726
I20260925 10:25:19.228509 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:19 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59220
I20260925 10:25:19.228981 14820 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f423b8d88fc42aeaf5637998a7b40bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:19 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:47254
I20260925 10:25:19.229485 14984 tablet_replica.cc:330] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: stopping tablet replica
I20260925 10:25:19.229696 14984 raft_consensus.cc:2229] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:19.230312 14984 raft_consensus.cc:2258] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:19.231683 14984 ts_tablet_manager.cc:1905] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:19.239977 14984 ts_tablet_manager.cc:1918] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:19.240134 14984 log.cc:1196] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/1f423b8d88fc42aeaf5637998a7b40bf
I20260925 10:25:19.240674 14984 ts_tablet_manager.cc:1939] T 1f423b8d88fc42aeaf5637998a7b40bf P 929c64ff70954b24b644b5b0c1f15f9e: Deleting consensus metadata
I20260925 10:25:19.241909 14393 catalog_manager.cc:4841] TS 929c64ff70954b24b644b5b0c1f15f9e (127.12.248.65:41653): tablet 1f423b8d88fc42aeaf5637998a7b40bf (table test-table [id=498687aed4d74f32b5decfadbc17f83a]) successfully deleted
I20260925 10:25:19.246039 14985 tablet_replica.cc:330] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: stopping tablet replica
I20260925 10:25:19.246459 14985 raft_consensus.cc:2229] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:19.247035 14985 raft_consensus.cc:2258] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:19.247821 14983 tablet_replica.cc:330] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: stopping tablet replica
I20260925 10:25:19.248024 14983 raft_consensus.cc:2229] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:19.248309 14983 raft_consensus.cc:2258] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:19.249297 14985 ts_tablet_manager.cc:1905] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:19.258395 14983 ts_tablet_manager.cc:1905] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:19.266512 14983 ts_tablet_manager.cc:1918] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:19.266652 14983 log.cc:1196] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/1f423b8d88fc42aeaf5637998a7b40bf
I20260925 10:25:19.267220 14985 ts_tablet_manager.cc:1918] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:19.267359 14985 log.cc:1196] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/1f423b8d88fc42aeaf5637998a7b40bf
I20260925 10:25:19.267851 14985 ts_tablet_manager.cc:1939] T 1f423b8d88fc42aeaf5637998a7b40bf P e34d36a006fd4d9bb0fccc0435103c9f: Deleting consensus metadata
I20260925 10:25:19.268237 14983 ts_tablet_manager.cc:1939] T 1f423b8d88fc42aeaf5637998a7b40bf P 1570a8edb8c242b3bf1654604507bb1b: Deleting consensus metadata
I20260925 10:25:19.269479 14392 catalog_manager.cc:4841] TS 1570a8edb8c242b3bf1654604507bb1b (127.12.248.66:35975): tablet 1f423b8d88fc42aeaf5637998a7b40bf (table test-table [id=498687aed4d74f32b5decfadbc17f83a]) successfully deleted
I20260925 10:25:19.270787 14391 catalog_manager.cc:4841] TS e34d36a006fd4d9bb0fccc0435103c9f (127.12.248.67:32821): tablet 1f423b8d88fc42aeaf5637998a7b40bf (table test-table [id=498687aed4d74f32b5decfadbc17f83a]) successfully deleted
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:19.283691 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.247158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.12.248.65:41653)
0925 10:25:19.247352 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.247358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.265420 (+ 18062us) server_negotiation.cc:407] Connection header received
0925 10:25:19.265808 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.265815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.265916 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.266000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:19.268573 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.269893 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.272083 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.272532 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.272691 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.276911 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:19.276925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:19.276947 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:19.276983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:19.280206 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.280992 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.281001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.281003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.281263 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.281738 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.281745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.281748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.282218 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:19.282542 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.283084 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.283338 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 10:25:19.284637 14608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59232
I20260925 10:25:19.301549 14950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.287457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.12.248.126:44581)
0925 10:25:19.287664 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.287670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.288463 (+ 793us) server_negotiation.cc:407] Connection header received
0925 10:25:19.288657 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.288662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.288741 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.288848 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:19.290151 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.290899 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.292472 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.293031 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.293173 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.295610 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:19.295615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:19.295619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:19.295651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:19.298936 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.299568 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.299574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.299578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.299715 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.300079 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.300084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.300088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.300383 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:19.300638 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.300928 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.301154 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 10:25:19.302327 14406 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40202
I20260925 10:25:19.304303 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14477
I20260925 10:25:19.320281 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.306239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.12.248.65:41653)
0925 10:25:19.306428 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.306433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.306447 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:25:19.306491 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.306494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.306552 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.306678 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:19.308339 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.309109 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.310377 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.310831 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.310967 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.313251 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:19.313256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:19.313259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:19.313289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:19.316934 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.317726 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.317736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.317739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.317958 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.318484 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.318491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.318496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.318926 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:19.319179 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.319513 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.319770 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 10:25:19.320955 14608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59246
W20260925 10:25:19.321578 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14477: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:19.321774 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14477
I20260925 10:25:19.346715 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14617
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:19.366384 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.347479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.12.248.66:35975)
0925 10:25:19.347702 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.347708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.347730 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:25:19.347913 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.347919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.348024 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.348133 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:19.349702 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.350796 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.352533 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.353214 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.353341 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.358959 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:19.358980 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:19.359002 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:19.359054 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:19.363144 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.363817 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.363827 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.363831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.364002 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.364448 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.364453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.364457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.364987 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:19.365383 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.365750 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.366010 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 10:25:19.367182 14747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35738
W20260925 10:25:19.368225 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:19.368410 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14617
I20260925 10:25:19.397635 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14756
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:19.415973 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:19.398427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.12.248.67:32821)
0925 10:25:19.398603 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:19.398608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:19.398625 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:25:19.398845 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:19.398849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:19.398907 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:19.398988 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:19.400607 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.401364 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.403293 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:19.403719 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:19.403838 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:19.409402 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:19.409417 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:19.409537 (+ 120us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:19.409604 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:19.413035 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.413918 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.413926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.413928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.414065 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:19.414485 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:19.414489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:19.414491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:19.414867 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:19.415122 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:19.415455 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 10:25:19.415625 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 10:25:19.416867 14886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47270
W20260925 10:25:19.417621 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:19.417801 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14756
I20260925 10:25:19.437996 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 14374
I20260925 10:25:19.439291 14446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:19.555939 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 14374
2026-09-25T10:25:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4954 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14994](info): set up 2 sockets
Sep 25 10:25:19 dist-test-slave-f9rl krb5kdc[14994](info): commencing operation
krb5kdc: starting...
W20260925 10:25:20.707302 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:20 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:20.902952 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34671
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:34671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:21.232297 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:21.232676 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:21.232851 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:21.241003 15007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:21.241139 15007 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:25:21.241240 15007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:21.241360 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:21.241465 15007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:21.241559 15007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:21.251784 15007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:34671
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:21.253172 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:21.254678 15007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:21.266799 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:21.266848 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:21.266821 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:21.267639 15007 server_base.cc:1034] running on GCE node
I20260925 10:25:21.268412 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:21.269887 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:21.271102 15007 hybrid_clock.cc:648] HybridClock initialized: now 1790331921271045 us; error 28 us; skew 500 ppm
I20260925 10:25:21.271495 15007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:21 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:21.274693 15007 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:21.276073 15007 webserver.cc:460] Webserver started at http://127.12.248.126:44445/ using document root <none> and password file <none>
I20260925 10:25:21.276767 15007 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:21.276896 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:21.277216 15007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:21.280143 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "5bf4f142c9894edf87ca529232549f99"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26c3e19acbf87748a1994c7576dbf8fc"
server_key_iv: "f4eac3786edab7089fbdcb4dcf12879a"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.280908 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "5bf4f142c9894edf87ca529232549f99"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26c3e19acbf87748a1994c7576dbf8fc"
server_key_iv: "f4eac3786edab7089fbdcb4dcf12879a"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.286027 15007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:25:21.289371 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:21.290719 15007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:21.290910 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "5bf4f142c9894edf87ca529232549f99"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26c3e19acbf87748a1994c7576dbf8fc"
server_key_iv: "f4eac3786edab7089fbdcb4dcf12879a"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.291127 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:21.317178 15007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:21.321676 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:21.321972 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:21.344174 15007 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:34671
I20260925 10:25:21.344193 15083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:34671 every 8 connection(s)
I20260925 10:25:21.345649 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:21.349512 15084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:21.351322 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15007
I20260925 10:25:21.351544 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:21.352196 13281 external_mini_cluster.cc:1428] Setting key 0ce9cbb0e1d25d628bb3665f5cf1d2d6
I20260925 10:25:21.362795 15084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Bootstrap starting.
I20260925 10:25:21.366441 15084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:21.367623 15084 log.cc:824] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:25:21 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:21.370481 15084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: No bootstrap required, opened a new log
I20260925 10:25:21.377250 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:21.354375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.12.248.126:34671)
0925 10:25:21.355242 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:21.355257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:21.355316 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:25:21.357788 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:21.357819 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:21.358324 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:21.358800 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:21.360086 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.362090 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.363401 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.364424 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.364642 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:21.368937 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:21.368944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:21.368958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:21.368995 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:21.372701 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.373394 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.373402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.373410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.373643 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.374007 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.374013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.374022 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.374713 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:21.375015 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:21.375392 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 10:25:21.375841 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":372,"threads_started":1}
I20260925 10:25:21.380388 15084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } }
I20260925 10:25:21.380710 15084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:21.380787 15084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf4f142c9894edf87ca529232549f99, State: Initialized, Role: FOLLOWER
I20260925 10:25:21.381270 15084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } }
I20260925 10:25:21.381551 15084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:21.381734 15084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:21.381930 15084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:21.384101 15084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } }
I20260925 10:25:21.384550 15084 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf4f142c9894edf87ca529232549f99; no voters:
I20260925 10:25:21.385087 15084 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:21.385313 15089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:21.386181 15089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf4f142c9894edf87ca529232549f99, State: Running, Role: LEADER
I20260925 10:25:21.386859 15089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } }
I20260925 10:25:21.387259 15084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:21.390064 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf4f142c9894edf87ca529232549f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } } }
I20260925 10:25:21.390527 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:21.390937 15091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf4f142c9894edf87ca529232549f99. Latest consensus state: current_term: 1 leader_uuid: "5bf4f142c9894edf87ca529232549f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf4f142c9894edf87ca529232549f99" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34671 } } }
I20260925 10:25:21.391302 15091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:21.392292 15097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:21.398402 15097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:21.407591 15097 catalog_manager.cc:1347] Generated new cluster ID: bcb18685ebd54a748baddf413e7a48df
I20260925 10:25:21.407735 15097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:21.441291 15097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:21.443251 15097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:21.453285 15097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Generated new TSK 0
I20260925 10:25:21.454422 15097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:21.512944 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34671
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:21.838652 15110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:21.838935 15110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:21.839033 15110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:21.846206 15110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:21.846325 15110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:21.846412 15110 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:25:21.846465 15110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:21.846526 15110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:21.856118 15110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:21.857472 15110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:21.858780 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:21.872561 15115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:21.872610 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:21.872642 15116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:21.873368 15110 server_base.cc:1034] running on GCE node
I20260925 10:25:21.874106 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:21.875586 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:21.876843 15110 hybrid_clock.cc:648] HybridClock initialized: now 1790331921876775 us; error 63 us; skew 500 ppm
I20260925 10:25:21.877784 15110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:21 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:21.881111 15110 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:21.882553 15110 webserver.cc:460] Webserver started at http://127.12.248.65:45609/ using document root <none> and password file <none>
I20260925 10:25:21.883219 15110 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:21.883324 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:21.883586 15110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:21.886667 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "ae36637e147c4564956090cf8697dad1"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26422f11d7caf3ad76930ea1580d6d8b"
server_key_iv: "22f35cf6a173e41b4d192be70bd69cce"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.887413 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "ae36637e147c4564956090cf8697dad1"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26422f11d7caf3ad76930ea1580d6d8b"
server_key_iv: "22f35cf6a173e41b4d192be70bd69cce"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.893301 15110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:25:21.896920 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:21.898384 15110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:21.898576 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "ae36637e147c4564956090cf8697dad1"
format_stamp: "Formatted at 2026-09-25 10:25:21 on dist-test-slave-f9rl"
server_key: "26422f11d7caf3ad76930ea1580d6d8b"
server_key_iv: "22f35cf6a173e41b4d192be70bd69cce"
server_key_version: "encryptionkey@0"
I20260925 10:25:21.898800 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:21.914980 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:21.919353 15110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:21.919657 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:21.922343 15110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:21.922492 15110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:21.922621 15110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:21.922729 15110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:25:21 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:21.950053 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:21.923927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.65:53351)
0925 10:25:21.924801 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:21.924848 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:21.927468 (+ 2620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:21.928330 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:21.928345 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:21.929183 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:21.930224 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:21.930254 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.931827 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.931831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:21.934016 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:21.934037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.935130 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.935142 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:21.935485 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:21.936391 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:21.936412 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:21.940181 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:21.944250 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:21.944265 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:21.944290 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:21.945164 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:21.945682 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:21.945694 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:21.945705 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:21.946077 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:21.946841 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:21.946847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:21.947419 (+ 572us) client_negotiation.cc:769] Sending connection context
0925 10:25:21.947743 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 10:25:21.948386 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":393,"threads_started":1}
I20260925 10:25:21.950506 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:21.923302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:53351 (local address 127.12.248.126:34671)
0925 10:25:21.923786 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:21.923792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:21.924986 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 10:25:21.927815 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:21.927821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:21.927949 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:21.928079 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:21.930428 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.931642 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.934230 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.934819 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.934950 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:21.940451 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:21.940457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:21.940462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:21.940509 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:21.943945 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.945347 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.945355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.945360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.945540 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.946220 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.946225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.946228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.946627 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:21.946781 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:21.949652 (+ 2871us) server_negotiation.cc:299] Negotiation successful
0925 10:25:21.949959 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260925 10:25:21.968863 15110 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:46859
I20260925 10:25:21.968892 15245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:46859 every 8 connection(s)
I20260925 10:25:21.970463 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:21.971544 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:21.956052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.65:40665)
0925 10:25:21.956792 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:21.956819 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:21.956988 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:21.957395 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:21.957401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:21.958089 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:21.958519 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:21.958539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.960023 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.960027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:21.961792 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:21.961812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.962754 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.962765 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:21.962942 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:21.963742 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:21.963761 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:21.965001 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:21.968674 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:21.968679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:21.968683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:21.969261 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:21.969676 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:21.969679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:21.969681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:21.969810 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:21.970394 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:21.970397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:21.970586 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 10:25:21.970737 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:25:21.971033 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":349,"threads_started":1}
I20260925 10:25:21.971607 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:21.956020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:40665 (local address 127.12.248.126:34671)
0925 10:25:21.956255 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:21.956262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:21.956919 (+ 657us) server_negotiation.cc:407] Connection header received
0925 10:25:21.957178 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:21.957183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:21.957245 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:21.957360 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:21.958728 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.959873 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.961956 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:21.962545 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:21.962680 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:21.965213 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:21.965218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:21.965221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:21.965256 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:21.968523 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.969379 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.969386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.969392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.969565 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:21.969917 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:21.969925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:21.969931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:21.970279 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:21.970441 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:21.970851 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 10:25:21.971144 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:25:21.972343 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15110
I20260925 10:25:21.972585 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:21.973415 15246 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34671
I20260925 10:25:21.973657 13281 external_mini_cluster.cc:1428] Setting key 0c68053bfde0d9875cb9248b722747a1
I20260925 10:25:21.973767 15246 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:21.974678 15246 heartbeater.cc:511] Master 127.12.248.126:34671 requested a full tablet report, sending...
I20260925 10:25:21.977329 15038 ts_manager.cc:194] Registered new tserver with Master: ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859)
I20260925 10:25:21.979483 15038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:53351
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:22.017295 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34671
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:22.346539 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:22.346807 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:22.346971 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:22.354506 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:22.354609 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:22.354748 15250 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:25:22.354805 15250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:22.354955 15250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:22.364907 15250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:22.366346 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:22.367715 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:22.381227 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:22.381302 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:22.381255 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:22.381942 15250 server_base.cc:1034] running on GCE node
I20260925 10:25:22.382506 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:22.383376 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:22.384577 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790331922384529 us; error 46 us; skew 500 ppm
I20260925 10:25:22.384902 15250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:22 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:22.394008 15250 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:22.395689 15250 webserver.cc:460] Webserver started at http://127.12.248.66:40245/ using document root <none> and password file <none>
I20260925 10:25:22.396427 15250 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:22.396577 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:22.396891 15250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:22.400032 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "74a6c5c2a0254d14a04896cb663183a3"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "f407fc10deac4455b11ea760bcb5a7ae"
server_key_iv: "daa29c60ae2d1f3e97f8b8a71da9baf2"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.400794 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "74a6c5c2a0254d14a04896cb663183a3"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "f407fc10deac4455b11ea760bcb5a7ae"
server_key_iv: "daa29c60ae2d1f3e97f8b8a71da9baf2"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.406032 15250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 10:25:22.409368 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:22.410694 15250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:22.410866 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "74a6c5c2a0254d14a04896cb663183a3"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "f407fc10deac4455b11ea760bcb5a7ae"
server_key_iv: "daa29c60ae2d1f3e97f8b8a71da9baf2"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.411072 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:22.427506 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:22.432078 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:22.432399 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:22.435189 15250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:22.435364 15250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:22.435564 15250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:22.435670 15250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:25:22 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:22.461637 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.436369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.66:37053)
0925 10:25:22.437282 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:22.437338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:22.440144 (+ 2806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:22.441087 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:22.441103 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:22.442133 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:22.443325 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.443357 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.444885 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.444893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.446606 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.446619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.447483 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.447491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.447721 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.448511 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:22.448532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:22.451422 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:22.455941 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.455956 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.455983 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.456657 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.457132 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.457143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.457149 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.457498 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.458319 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:22.458330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:22.458848 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 10:25:22.459224 (+ 376us) client_negotiation.cc:240] Negotiation successful
0925 10:25:22.459881 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":351,"threads_started":1}
I20260925 10:25:22.462533 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.435755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:37053 (local address 127.12.248.126:34671)
0925 10:25:22.435989 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:22.435999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:22.437503 (+ 1504us) server_negotiation.cc:407] Connection header received
0925 10:25:22.440492 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:22.440498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:22.440625 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:22.440774 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:22.443648 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.444651 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.446765 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.447222 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.447369 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.451780 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:22.451789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:22.451796 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:22.451841 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:22.455725 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.456811 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.456816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.456820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.456997 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.457698 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.457715 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.457719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.458162 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:22.458342 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:22.461701 (+ 3359us) server_negotiation.cc:299] Negotiation successful
0925 10:25:22.462040 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:25:22.483976 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.469045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:59247 (local address 127.12.248.126:34671)
0925 10:25:22.469268 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:22.469277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:22.470271 (+ 994us) server_negotiation.cc:407] Connection header received
0925 10:25:22.470508 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:22.470513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:22.470608 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:22.470750 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:22.472559 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.473693 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.475017 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.475431 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.475561 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.477703 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:22.477713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:22.477716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:22.477748 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:22.480935 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.481964 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.481971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.481974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.482119 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.482495 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.482500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.482503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.482836 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:22.482998 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:22.483397 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 10:25:22.483625 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:25:22.484231 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.469064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.66:59247)
0925 10:25:22.470150 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:22.470166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:22.470357 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:22.470787 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:22.470793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:22.471518 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:22.472335 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.472351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.473845 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.473850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.474870 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.474881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.475625 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.475633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.475803 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.476462 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:22.476509 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:22.477400 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:22.481091 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.481097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.481101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.481821 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.482236 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.482243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.482247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.482378 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.482961 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:22.482966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:22.483153 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 10:25:22.483292 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 10:25:22.483693 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":692,"threads_started":1}
I20260925 10:25:22.492149 15250 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:32773
I20260925 10:25:22.492187 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:32773 every 8 connection(s)
I20260925 10:25:22.493613 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:22.495733 15385 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34671
I20260925 10:25:22.495973 15385 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:22.496750 15385 heartbeater.cc:511] Master 127.12.248.126:34671 requested a full tablet report, sending...
I20260925 10:25:22.496865 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15250
I20260925 10:25:22.497053 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:22.497670 13281 external_mini_cluster.cc:1428] Setting key de2dd63af4866e7f9b348d4a969f8d84
I20260925 10:25:22.498463 15038 ts_manager.cc:194] Registered new tserver with Master: 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773)
I20260925 10:25:22.499579 15038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:37053
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:22.541164 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:34671
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:22.864328 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:22.864629 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:22.864734 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:22.872232 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:22.872347 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:22.872519 15389 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:25:22.872608 15389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:25:22.872748 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:22.884230 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:22.885623 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:22.886993 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:22.901289 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:22.901364 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:22.901368 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:22.902431 15389 server_base.cc:1034] running on GCE node
I20260925 10:25:22.903177 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:22.904412 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:22.905668 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790331922905604 us; error 68 us; skew 500 ppm
I20260925 10:25:22.906062 15389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:22 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:22.909411 15389 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:22.910898 15389 webserver.cc:460] Webserver started at http://127.12.248.67:33185/ using document root <none> and password file <none>
I20260925 10:25:22.911614 15389 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:22.911762 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:22.912086 15389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:22.915180 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "405ae85471230372b5e3689381a2e5fe"
server_key_iv: "5f4daa22894234cfd40a60f321ceb292"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.915990 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "405ae85471230372b5e3689381a2e5fe"
server_key_iv: "5f4daa22894234cfd40a60f321ceb292"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.921214 15389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 10:25:22.924573 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:22.926046 15389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:22.926281 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
format_stamp: "Formatted at 2026-09-25 10:25:22 on dist-test-slave-f9rl"
server_key: "405ae85471230372b5e3689381a2e5fe"
server_key_iv: "5f4daa22894234cfd40a60f321ceb292"
server_key_version: "encryptionkey@0"
I20260925 10:25:22.926488 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:22.939532 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:22.943555 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:22.944012 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:22.946892 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:22.947063 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:22.947256 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:22.947358 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:25:22 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331922, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:22.971138 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.947353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:41385 (local address 127.12.248.126:34671)
0925 10:25:22.947572 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:22.947590 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:22.949031 (+ 1441us) server_negotiation.cc:407] Connection header received
0925 10:25:22.951103 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:22.951109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:22.951207 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:22.951351 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:22.953832 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.954909 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.957553 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.957969 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.958106 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.963186 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:22.963195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:22.963202 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:22.963254 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:22.966617 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.967633 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.967638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.967642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.967819 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.968665 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.968670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.968674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.969026 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:22.969193 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:22.970522 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0925 10:25:22.970790 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:25:22.973070 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.947995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.67:41385)
0925 10:25:22.948838 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:22.948893 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:22.950824 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:22.951640 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:22.951651 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:22.952465 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:22.953556 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.953587 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.955072 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.955080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.957351 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.957364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.958391 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.958399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.958587 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.959547 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:22.959586 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:22.962829 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:22.966794 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.966801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.966814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.967499 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.967971 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.967980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.967986 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.968483 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.969197 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:22.969209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:22.969903 (+ 694us) client_negotiation.cc:769] Sending connection context
0925 10:25:22.970332 (+ 429us) client_negotiation.cc:240] Negotiation successful
0925 10:25:22.971088 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
I20260925 10:25:22.982507 15246 heartbeater.cc:503] Master 127.12.248.126:34671 was elected leader, sending a full tablet report...
I20260925 10:25:22.993475 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.979287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:48637 (local address 127.12.248.126:34671)
0925 10:25:22.979474 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:22.979479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:22.980180 (+ 701us) server_negotiation.cc:407] Connection header received
0925 10:25:22.980385 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:22.980389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:22.980448 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:22.980550 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:22.981632 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.982686 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.985378 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.985889 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.986031 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.987945 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:22.987949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:22.987953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:22.988001 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:22.990658 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.991372 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.991379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.991382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.991506 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:22.991861 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:22.991867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:22.991870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:22.992150 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:22.992303 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:22.992857 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 10:25:22.993072 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:25:22.993600 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:22.979423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.12.248.67:48637)
0925 10:25:22.980072 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:22.980088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:22.980245 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:22.980572 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:22.980577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:22.981044 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:22.981406 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.981416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.982862 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.982872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.985237 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:22.985250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:22.986078 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:22.986084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:22.986263 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:22.987009 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:22.987025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:22.987755 (+ 730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:22.990846 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.990851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.990855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.991254 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.991618 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:22.991623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:22.991627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:22.991754 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:22.992284 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:22.992292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:22.992529 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 10:25:22.992757 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 10:25:22.993109 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":398,"threads_started":1}
I20260925 10:25:23.007153 15389 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:38127
I20260925 10:25:23.007191 15523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:38127 every 8 connection(s)
I20260925 10:25:23.008405 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:23.009425 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15389
I20260925 10:25:23.009622 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:23.010139 13281 external_mini_cluster.cc:1428] Setting key 6a70c27e5b0929589fc942b9ab88cfd4
I20260925 10:25:23.010879 15524 heartbeater.cc:347] Connected to a master server at 127.12.248.126:34671
I20260925 10:25:23.011138 15524 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:23.011991 15524 heartbeater.cc:511] Master 127.12.248.126:34671 requested a full tablet report, sending...
I20260925 10:25:23.013564 15038 ts_manager.cc:194] Registered new tserver with Master: ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:23.014398 15038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:41385
I20260925 10:25:23.025594 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:25:23 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:25:23 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:23.073577 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.059975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.12.248.126:34671)
0925 10:25:23.060209 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.060217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.060417 (+ 200us) server_negotiation.cc:407] Connection header received
0925 10:25:23.060579 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.060583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.060648 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.060750 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:23.062195 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.062934 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.064207 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.064764 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.064887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.068743 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:23.068748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:23.068751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:23.068783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:23.071276 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:23.071865 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:23.071868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:23.071871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:23.071985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:23.072306 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:23.072309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:23.072311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:23.072566 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:23.072753 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.072992 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.073157 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 10:25:23.077791 15038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:23.084642 15038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:23.119213 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.101800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.0.0.1:38350)
0925 10:25:23.104372 (+ 2572us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.104401 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.104533 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.107046 (+ 2513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.107057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.107101 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.107681 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.107699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.110862 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.110870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.114352 (+ 3482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.114376 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.117992 (+ 3616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.118000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.118318 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.118389 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.118503 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.118742 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2411,"mutex_wait_us":1,"thread_start_us":902,"threads_started":1}
I20260925 10:25:23.120399 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.102510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.12.248.67:38127)
0925 10:25:23.106175 (+ 3665us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.106189 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.106242 (+ 53us) server_negotiation.cc:407] Connection header received
0925 10:25:23.106371 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.106383 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.106740 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.107982 (+ 1242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.108376 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.110652 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.114669 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.116912 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.119573 (+ 2661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.119840 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.119906 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.120079 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3429,"thread_start_us":3213,"threads_started":1}
I20260925 10:25:23.124176 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.101541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.12.248.66:32773)
0925 10:25:23.117210 (+ 15669us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.117217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.117243 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:25:23.117326 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.117332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.117616 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.117845 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.118540 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.120098 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.122214 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.123081 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.123244 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.123453 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.123725 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.123892 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15515,"thread_start_us":15386,"threads_started":1}
I20260925 10:25:23.124223 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.101303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:32773 (local address 127.0.0.1:56346)
0925 10:25:23.102696 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.102755 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.103058 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.117822 (+ 14764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.117830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.117857 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.118335 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.118348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.120251 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.120256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.122054 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.122069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.123308 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.123315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.123533 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.123571 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.123672 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.123802 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":52,"thread_start_us":15905,"threads_started":1}
I20260925 10:25:23.124752 15454 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2da85be97d457781c7fd84989de45c (DEFAULT_TABLE table=test-table [id=308c987a42524cfa8967749498397459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.126451 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.103670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:46859 (local address 127.0.0.1:41738)
0925 10:25:23.117517 (+ 13847us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.117533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.117720 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.118392 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.118397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.118421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.118871 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.118890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.121832 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.121836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.123977 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.123993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.125887 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.125892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.126011 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.126034 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.126096 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.126194 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13690,"mutex_wait_us":3,"thread_start_us":2276,"threads_started":1}
I20260925 10:25:23.126469 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2da85be97d457781c7fd84989de45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.126629 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.104302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.12.248.65:46859)
0925 10:25:23.116525 (+ 12223us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.116540 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.117651 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 10:25:23.117923 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.117931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.118157 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.118398 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.119113 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.121568 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.124168 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.125689 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.125865 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.126098 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.126167 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.126325 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11980,"thread_start_us":12243,"threads_started":1}
I20260925 10:25:23.130123 15315 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2da85be97d457781c7fd84989de45c (DEFAULT_TABLE table=test-table [id=308c987a42524cfa8967749498397459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.130275 15176 tablet_service.cc:1467] Processing CreateTablet for tablet 3f2da85be97d457781c7fd84989de45c (DEFAULT_TABLE table=test-table [id=308c987a42524cfa8967749498397459]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.131639 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2da85be97d457781c7fd84989de45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.131650 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f2da85be97d457781c7fd84989de45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.141978 15543 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Bootstrap starting.
I20260925 10:25:23.145603 15545 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Bootstrap starting.
I20260925 10:25:23.146591 15544 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Bootstrap starting.
I20260925 10:25:23.147377 15543 tablet_bootstrap.cc:654] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.149195 15543 log.cc:824] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:23.149567 15545 tablet_bootstrap.cc:654] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.150059 15544 tablet_bootstrap.cc:654] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.150861 15545 log.cc:824] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:23.151207 15544 log.cc:824] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:23.156093 15545 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: No bootstrap required, opened a new log
I20260925 10:25:23.156209 15543 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: No bootstrap required, opened a new log
I20260925 10:25:23.156464 15545 ts_tablet_manager.cc:1397] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260925 10:25:23.156495 15543 ts_tablet_manager.cc:1397] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260925 10:25:23.157196 15544 tablet_bootstrap.cc:492] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: No bootstrap required, opened a new log
I20260925 10:25:23.157500 15544 ts_tablet_manager.cc:1397] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 10:25:23.164196 15545 raft_consensus.cc:348] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.164675 15545 raft_consensus.cc:374] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.164804 15545 raft_consensus.cc:729] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36637e147c4564956090cf8697dad1, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.165486 15545 consensus_queue.cc:260] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.167434 15545 ts_tablet_manager.cc:1428] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260925 10:25:23.167428 15543 raft_consensus.cc:348] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.167866 15543 raft_consensus.cc:374] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.168011 15543 raft_consensus.cc:729] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.168076 15544 raft_consensus.cc:348] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.168493 15544 raft_consensus.cc:374] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.168592 15544 raft_consensus.cc:729] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.168638 15543 consensus_queue.cc:260] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.169245 15544 consensus_queue.cc:260] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.169833 15524 heartbeater.cc:503] Master 127.12.248.126:34671 was elected leader, sending a full tablet report...
I20260925 10:25:23.170463 15543 ts_tablet_manager.cc:1428] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260925 10:25:23.172079 15385 heartbeater.cc:503] Master 127.12.248.126:34671 was elected leader, sending a full tablet report...
I20260925 10:25:23.172435 15544 ts_tablet_manager.cc:1428] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260925 10:25:23.227166 15551 raft_consensus.cc:482] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:23.227563 15551 raft_consensus.cc:504] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.229246 15551 leader_election.cc:288] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127), ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859)
W20260925 10:25:23.231554 15247 tablet.cc:2369] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:23.236866 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.229671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:54669 (local address 127.12.248.67:38127)
0925 10:25:23.229875 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.229879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.230481 (+ 602us) server_negotiation.cc:407] Connection header received
0925 10:25:23.230734 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.230741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.230841 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.230949 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.231656 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.232770 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.234834 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.236033 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.236150 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.236203 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.236585 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.236677 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:25:23.237061 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.229614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.12.248.66:54669)
0925 10:25:23.230335 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.230354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.230509 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.231014 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.231021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.231045 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.231490 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.231505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.232901 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.232907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.234670 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.234682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.236241 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.236249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.236420 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.236455 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.236558 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.236708 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":342,"threads_started":1}
I20260925 10:25:23.238039 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.230844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:38621 (local address 127.12.248.65:46859)
0925 10:25:23.231238 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.231244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.231332 (+ 88us) server_negotiation.cc:407] Connection header received
0925 10:25:23.231531 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.231539 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.231623 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.231714 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.232684 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.233812 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.235937 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.237216 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.237387 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.237498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.237761 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.237874 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260925 10:25:23.238178 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.230533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:46859 (local address 127.12.248.66:38621)
0925 10:25:23.231212 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.231230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.231375 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.231880 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.231884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.232133 (+ 249us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.232504 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.232516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.233955 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.233959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.235768 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.235783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.237470 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.237479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.237612 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.237635 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.237706 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.237866 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":362,"threads_started":1}
I20260925 10:25:23.238353 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2da85be97d457781c7fd84989de45c" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" is_pre_election: true
I20260925 10:25:23.238904 15479 raft_consensus.cc:2454] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 0.
I20260925 10:25:23.239188 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2da85be97d457781c7fd84989de45c" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1" is_pre_election: true
I20260925 10:25:23.239606 15266 leader_election.cc:302] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.239715 15196 raft_consensus.cc:2454] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 0.
I20260925 10:25:23.240043 15551 raft_consensus.cc:2781] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:23.240204 15551 raft_consensus.cc:482] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:23.240345 15551 raft_consensus.cc:3037] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.242654 15551 raft_consensus.cc:504] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.243234 15551 leader_election.cc:288] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 election: Requested vote from peers ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127), ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859)
I20260925 10:25:23.243759 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2da85be97d457781c7fd84989de45c" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
I20260925 10:25:23.243831 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f2da85be97d457781c7fd84989de45c" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1"
I20260925 10:25:23.243969 15479 raft_consensus.cc:3037] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.244170 15196 raft_consensus.cc:3037] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.246358 15479 raft_consensus.cc:2454] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 1.
I20260925 10:25:23.246522 15196 raft_consensus.cc:2454] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 1.
I20260925 10:25:23.246810 15266 leader_election.cc:302] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.247125 15551 raft_consensus.cc:2781] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:23.247641 15551 raft_consensus.cc:686] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 1 LEADER]: Becoming Leader. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Running, Role: LEADER
W20260925 10:25:23.247749 15386 tablet.cc:2369] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:23.248194 15551 consensus_queue.cc:237] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } }
I20260925 10:25:23.252647 15037 catalog_manager.cc:5495] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "74a6c5c2a0254d14a04896cb663183a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } health_report { overall_health: UNKNOWN } } }
W20260925 10:25:23.260085 15525 tablet.cc:2369] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:25:23 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331921, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:23.301838 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.288087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:34671 (local address 127.0.0.1:56460)
0925 10:25:23.288663 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.288677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.288860 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.289193 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.289197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.290001 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:23.290282 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.290292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.291310 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.291318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.292286 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.292296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.293142 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.293150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.293292 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.294037 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:23.294054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:23.296348 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:23.299212 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:23.299227 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:23.299242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:23.299627 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:23.299991 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:23.299995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:23.299999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:23.300191 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:23.300704 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:23.300713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:23.300885 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.301045 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.301346 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":334,"threads_started":1}
I20260925 10:25:23.301843 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.287992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.12.248.126:34671)
0925 10:25:23.288186 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.288191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.288777 (+ 586us) server_negotiation.cc:407] Connection header received
0925 10:25:23.288983 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.288988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.289055 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.289158 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:23.290423 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.291183 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.292428 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.292947 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.293079 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.296531 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:23.296537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:23.296541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:23.296581 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:23.299083 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:23.299751 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:23.299756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:23.299760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:23.299877 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:23.300314 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:23.300318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:23.300322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:23.300579 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:23.300747 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.301138 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.301511 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":75}
I20260925 10:25:23.307741 15037 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56460:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:25:23.308727 15037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:23.315621 15176 tablet_service.cc:1467] Processing CreateTablet for tablet f0e4fe18c4cb492c9b2578314c1db8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33a546f4bad848f2944163e7f8692e72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:23.315793 15315 tablet_service.cc:1467] Processing CreateTablet for tablet f0e4fe18c4cb492c9b2578314c1db8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33a546f4bad848f2944163e7f8692e72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:23.316006 15454 tablet_service.cc:1467] Processing CreateTablet for tablet f0e4fe18c4cb492c9b2578314c1db8eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33a546f4bad848f2944163e7f8692e72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:25:23.316666 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e4fe18c4cb492c9b2578314c1db8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.316697 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e4fe18c4cb492c9b2578314c1db8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.316725 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e4fe18c4cb492c9b2578314c1db8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.322542 15545 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: Bootstrap starting.
I20260925 10:25:23.323707 15544 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: Bootstrap starting.
I20260925 10:25:23.323987 15545 tablet_bootstrap.cc:654] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.325547 15544 tablet_bootstrap.cc:654] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.325693 15543 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: Bootstrap starting.
I20260925 10:25:23.327026 15543 tablet_bootstrap.cc:654] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.328397 15544 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: No bootstrap required, opened a new log
I20260925 10:25:23.328600 15544 ts_tablet_manager.cc:1397] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260925 10:25:23.329530 15543 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: No bootstrap required, opened a new log
I20260925 10:25:23.329644 15545 tablet_bootstrap.cc:492] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: No bootstrap required, opened a new log
I20260925 10:25:23.329492 15544 raft_consensus.cc:348] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.329682 15543 ts_tablet_manager.cc:1397] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:25:23.329734 15544 raft_consensus.cc:374] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.329838 15545 ts_tablet_manager.cc:1397] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260925 10:25:23.329852 15544 raft_consensus.cc:729] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.330092 15544 consensus_queue.cc:260] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.330410 15543 raft_consensus.cc:348] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.330652 15543 raft_consensus.cc:374] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.330740 15543 raft_consensus.cc:729] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.330588 15545 raft_consensus.cc:348] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.330574 15551 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.330825 15545 raft_consensus.cc:374] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.330891 15551 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.330914 15545 raft_consensus.cc:729] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36637e147c4564956090cf8697dad1, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.330956 15543 consensus_queue.cc:260] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.331593 15543 ts_tablet_manager.cc:1428] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:25:23.331422 15550 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.331808 15550 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.331168 15545 consensus_queue.cc:260] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.332317 15544 ts_tablet_manager.cc:1428] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 10:25:23.332794 15545 ts_tablet_manager.cc:1428] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:23.332918 15549 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.333325 15549 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.344304 15550 raft_consensus.cc:482] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:23.344525 15550 raft_consensus.cc:504] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.346383 15550 leader_election.cc:288] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859), 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773)
I20260925 10:25:23.353503 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.346867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:52253 (local address 127.12.248.66:32773)
0925 10:25:23.347070 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.347078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.347761 (+ 683us) server_negotiation.cc:407] Connection header received
0925 10:25:23.347836 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.347841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.347919 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.348005 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.348592 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.349739 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.351551 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.352419 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.352587 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.352664 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.353101 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.353224 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:25:23.353552 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.347282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:32773 (local address 127.12.248.67:52253)
0925 10:25:23.347493 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.347513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.347633 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.348179 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.348184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.348213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.348438 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.348449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.349871 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.349876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.351395 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.351407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.352566 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.352572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.352914 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.352963 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.353055 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.353188 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":63}
I20260925 10:25:23.354526 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e4fe18c4cb492c9b2578314c1db8eb" candidate_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3" is_pre_election: true
I20260925 10:25:23.354988 15340 raft_consensus.cc:2454] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee44c88e6c1b4d6ebfca31df2125c6c4 in term 0.
I20260925 10:25:23.355360 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.346747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:57505 (local address 127.12.248.65:46859)
0925 10:25:23.346953 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.346960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.348516 (+ 1556us) server_negotiation.cc:407] Connection header received
0925 10:25:23.348718 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.348723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.348783 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.348882 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:23.349413 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.350470 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.353023 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.354289 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.354437 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.354502 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.354933 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.355079 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:25:23.355746 15407 leader_election.cc:302] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.356186 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.347591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:46859 (local address 127.12.248.67:57505)
0925 10:25:23.348403 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.348417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.348563 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.348899 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.348903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.348940 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:23.349248 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.349256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.350611 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.350616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.352869 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.352884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.354535 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.354544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.354727 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.354769 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.355469 (+ 700us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.355609 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":43,"thread_start_us":471,"threads_started":1}
I20260925 10:25:23.356308 15550 raft_consensus.cc:2781] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:23.356550 15550 raft_consensus.cc:482] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:23.356662 15550 raft_consensus.cc:3037] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.357084 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e4fe18c4cb492c9b2578314c1db8eb" candidate_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1" is_pre_election: true
I20260925 10:25:23.357378 15196 raft_consensus.cc:2454] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee44c88e6c1b4d6ebfca31df2125c6c4 in term 0.
I20260925 10:25:23.359246 15550 raft_consensus.cc:504] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.359858 15550 leader_election.cc:288] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [CANDIDATE]: Term 1 election: Requested vote from peers ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859), 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773)
I20260925 10:25:23.360318 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e4fe18c4cb492c9b2578314c1db8eb" candidate_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1"
I20260925 10:25:23.360404 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e4fe18c4cb492c9b2578314c1db8eb" candidate_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3"
I20260925 10:25:23.360544 15196 raft_consensus.cc:3037] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.360599 15340 raft_consensus.cc:3037] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.362635 15196 raft_consensus.cc:2454] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee44c88e6c1b4d6ebfca31df2125c6c4 in term 1.
I20260925 10:25:23.363071 15405 leader_election.cc:302] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae36637e147c4564956090cf8697dad1, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.363363 15550 raft_consensus.cc:2781] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:23.363721 15340 raft_consensus.cc:2454] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee44c88e6c1b4d6ebfca31df2125c6c4 in term 1.
I20260925 10:25:23.363948 15550 raft_consensus.cc:686] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 LEADER]: Becoming Leader. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Running, Role: LEADER
I20260925 10:25:23.364782 15550 consensus_queue.cc:237] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.368700 15566 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: TxnStatusTablet state changed. Reason: New leader ee44c88e6c1b4d6ebfca31df2125c6c4. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.369032 15566 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:23.369763 15568 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:25:23.370023 15568 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:25:23.370635 15038 catalog_manager.cc:5495] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } health_report { overall_health: HEALTHY } } }
I20260925 10:25:23.393800 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.388039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.0.0.1:41824)
0925 10:25:23.388321 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.388337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.388497 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.388984 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.388989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.389003 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:23.389193 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.389202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.390322 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.390327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.391331 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.391341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.392091 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.392096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.392285 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.392336 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.393213 (+ 877us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.393244 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.393324 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.393505 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:25:23.393800 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.387970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.12.248.67:38127)
0925 10:25:23.388151 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.388158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.388409 (+ 251us) server_negotiation.cc:407] Connection header received
0925 10:25:23.388696 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.388700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.388801 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.388949 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:23.389348 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.390186 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.391483 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.391932 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.392069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.392526 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.393095 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.393236 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.393345 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.393427 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:25:23.402753 15340 raft_consensus.cc:1261] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Refusing update from remote peer ee44c88e6c1b4d6ebfca31df2125c6c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:23.402789 15196 raft_consensus.cc:1261] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Refusing update from remote peer ee44c88e6c1b4d6ebfca31df2125c6c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:23.403803 15566 consensus_queue.cc:1035] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:23.404089 15550 consensus_queue.cc:1035] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:23.419236 15549 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: TxnStatusTablet state changed. Reason: New leader ee44c88e6c1b4d6ebfca31df2125c6c4. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.419564 15549 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.422291 15571 mvcc.cc:205] Tried to move back new op lower bound from 7333199558246522880 to 7333199558107107328. Current Snapshot: MvccSnapshot[applied={T|T < 7333199558246522880}]
I20260925 10:25:23.422614 15550 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.422886 15550 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:23.423751 15551 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: TxnStatusTablet state changed. Reason: New leader ee44c88e6c1b4d6ebfca31df2125c6c4. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.424044 15551 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.426435 15550 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.426657 15550 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ee44c88e6c1b4d6ebfca31df2125c6c4: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:25:23.431954 15549 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.432222 15549 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P ae36637e147c4564956090cf8697dad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.434389 15551 tablet_replica.cc:439] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } } }
I20260925 10:25:23.434696 15551 tablet_replica.cc:442] T f0e4fe18c4cb492c9b2578314c1db8eb P 74a6c5c2a0254d14a04896cb663183a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:25:23.443509 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.437531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.12.248.66:32773)
0925 10:25:23.437694 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.437700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.437822 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:25:23.438108 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.438114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.438224 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.438377 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:23.438697 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.439802 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.441107 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.441763 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.441902 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.442258 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.442917 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.443064 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.443113 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.443196 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:25:23.460099 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.453915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:39611 (local address 127.12.248.67:38127)
0925 10:25:23.454146 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.454155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.454767 (+ 612us) server_negotiation.cc:407] Connection header received
0925 10:25:23.454959 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.454963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.455027 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.455100 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:23.455647 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.456408 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.458457 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.458872 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.458967 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.459496 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.459589 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.459655 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.459807 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.459871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:25:23.460319 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.453931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.12.248.66:39611)
0925 10:25:23.454659 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.454679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.454830 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.455139 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.455144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.455158 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:23.455503 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.455515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.456525 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.456531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.458320 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.458338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.459045 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.459052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.459241 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.459308 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.459687 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.459698 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.459782 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.459927 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260925 10:25:23.470913 15196 raft_consensus.cc:1261] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Refusing update from remote peer 74a6c5c2a0254d14a04896cb663183a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:23.471266 15479 raft_consensus.cc:1261] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Refusing update from remote peer 74a6c5c2a0254d14a04896cb663183a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:23.471553 15554 consensus_queue.cc:1035] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:23.471904 15551 consensus_queue.cc:1035] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:23.478758 15573 mvcc.cc:205] Tried to move back new op lower bound from 7333199558530527232 to 7333199557628026880. Current Snapshot: MvccSnapshot[applied={T|T < 7333199558530527232}]
I20260925 10:25:23.516515 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.508133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:51945 (local address 127.12.248.66:32773)
0925 10:25:23.508330 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:23.508336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:23.508566 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:25:23.508863 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:23.508868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:23.508946 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:23.509006 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:23.509795 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.510900 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.514113 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.514729 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.514838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.515729 (+ 891us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.515856 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.515922 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:23.516163 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:25:23.516262 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:25:23.521821 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:23.507841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:32773 (local address 127.12.248.67:51945)
0925 10:25:23.508475 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:23.508495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:23.508688 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:23.509226 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:23.509230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:23.509245 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:23.509610 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.509624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.511809 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.511815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.513736 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:23.513749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:23.515249 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:23.515257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:23.515445 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:23.515525 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:23.516013 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:23.516026 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:25:23.516768 (+ 742us) client_negotiation.cc:240] Negotiation successful
0925 10:25:23.516928 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":2,"thread_start_us":5015,"threads_started":1}
I20260925 10:25:23.569028 15038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:23.569345 15038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:23.573643 15038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Sending DeleteTablet for 3 replicas of tablet 3f2da85be97d457781c7fd84989de45c
I20260925 10:25:23.574795 15315 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2da85be97d457781c7fd84989de45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:23 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56346
I20260925 10:25:23.577423 15592 tablet_replica.cc:330] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: stopping tablet replica
I20260925 10:25:23.577889 15592 raft_consensus.cc:2229] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:23.578555 15592 raft_consensus.cc:2258] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:23.576257 15176 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2da85be97d457781c7fd84989de45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:23 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41738
I20260925 10:25:23.580693 15592 ts_tablet_manager.cc:1905] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:23.581914 15593 tablet_replica.cc:330] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: stopping tablet replica
I20260925 10:25:23.584472 15454 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f2da85be97d457781c7fd84989de45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:23 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:38350
I20260925 10:25:23.586644 15038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:23.587409 15038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:23.589607 15592 ts_tablet_manager.cc:1918] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:23.589699 15593 raft_consensus.cc:2229] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:23.589833 15592 log.cc:1196] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/3f2da85be97d457781c7fd84989de45c
I20260925 10:25:23.590080 15593 raft_consensus.cc:2258] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:23.593550 15593 ts_tablet_manager.cc:1905] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:23.597039 15594 tablet_replica.cc:330] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: stopping tablet replica
I20260925 10:25:23.597507 15594 raft_consensus.cc:2229] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:23.597843 15594 raft_consensus.cc:2258] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:23.597743 15315 tablet_service.cc:1467] Processing CreateTablet for tablet 087f851497a5471983c7620615e499b5 (DEFAULT_TABLE table=test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.598203 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087f851497a5471983c7620615e499b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.598467 15592 ts_tablet_manager.cc:1939] T 3f2da85be97d457781c7fd84989de45c P 74a6c5c2a0254d14a04896cb663183a3: Deleting consensus metadata
I20260925 10:25:23.600358 15025 catalog_manager.cc:4841] TS 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773): tablet 3f2da85be97d457781c7fd84989de45c (table test-table [id=308c987a42524cfa8967749498397459]) successfully deleted
I20260925 10:25:23.601984 15176 tablet_service.cc:1467] Processing CreateTablet for tablet 087f851497a5471983c7620615e499b5 (DEFAULT_TABLE table=test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.602421 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087f851497a5471983c7620615e499b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.606426 15593 ts_tablet_manager.cc:1918] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:23.606626 15593 log.cc:1196] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/3f2da85be97d457781c7fd84989de45c
I20260925 10:25:23.607270 15593 ts_tablet_manager.cc:1939] T 3f2da85be97d457781c7fd84989de45c P ae36637e147c4564956090cf8697dad1: Deleting consensus metadata
I20260925 10:25:23.609227 15453 tablet_service.cc:1467] Processing CreateTablet for tablet 087f851497a5471983c7620615e499b5 (DEFAULT_TABLE table=test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:23.609808 15453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087f851497a5471983c7620615e499b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:23.611907 15594 ts_tablet_manager.cc:1905] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:23.614567 15023 catalog_manager.cc:4841] TS ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859): tablet 3f2da85be97d457781c7fd84989de45c (table test-table [id=308c987a42524cfa8967749498397459]) successfully deleted
I20260925 10:25:23.615126 15544 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Bootstrap starting.
I20260925 10:25:23.617008 15544 tablet_bootstrap.cc:654] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.617394 15545 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Bootstrap starting.
I20260925 10:25:23.619081 15545 tablet_bootstrap.cc:654] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.623803 15543 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Bootstrap starting.
I20260925 10:25:23.625662 15543 tablet_bootstrap.cc:654] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:23.632382 15543 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: No bootstrap required, opened a new log
I20260925 10:25:23.632637 15543 ts_tablet_manager.cc:1397] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260925 10:25:23.633553 15543 raft_consensus.cc:348] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.634040 15543 raft_consensus.cc:374] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.634179 15543 raft_consensus.cc:729] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.634516 15543 consensus_queue.cc:260] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.635267 15543 ts_tablet_manager.cc:1428] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:25:23.636135 15545 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: No bootstrap required, opened a new log
I20260925 10:25:23.636307 15545 ts_tablet_manager.cc:1397] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20260925 10:25:23.637037 15545 raft_consensus.cc:348] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.637271 15545 raft_consensus.cc:374] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.637338 15545 raft_consensus.cc:729] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36637e147c4564956090cf8697dad1, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.637636 15545 consensus_queue.cc:260] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.641963 15545 ts_tablet_manager.cc:1428] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260925 10:25:23.644352 15594 ts_tablet_manager.cc:1918] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:23.644520 15594 log.cc:1196] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/3f2da85be97d457781c7fd84989de45c
I20260925 10:25:23.645108 15594 ts_tablet_manager.cc:1939] T 3f2da85be97d457781c7fd84989de45c P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting consensus metadata
I20260925 10:25:23.646929 15023 catalog_manager.cc:4841] TS ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127): tablet 3f2da85be97d457781c7fd84989de45c (table test-table [id=308c987a42524cfa8967749498397459]) successfully deleted
I20260925 10:25:23.647523 15544 tablet_bootstrap.cc:492] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: No bootstrap required, opened a new log
I20260925 10:25:23.647730 15544 ts_tablet_manager.cc:1397] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.004s
I20260925 10:25:23.648550 15544 raft_consensus.cc:348] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.648823 15544 raft_consensus.cc:374] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:23.648907 15544 raft_consensus.cc:729] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Initialized, Role: FOLLOWER
I20260925 10:25:23.649142 15544 consensus_queue.cc:260] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.651537 15544 ts_tablet_manager.cc:1428] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 10:25:23.912139 15554 raft_consensus.cc:482] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:23.912376 15554 raft_consensus.cc:504] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.913065 15554 leader_election.cc:288] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:23.913795 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087f851497a5471983c7620615e499b5" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" is_pre_election: true
I20260925 10:25:23.913867 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087f851497a5471983c7620615e499b5" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1" is_pre_election: true
I20260925 10:25:23.914057 15479 raft_consensus.cc:2454] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 0.
I20260925 10:25:23.914224 15196 raft_consensus.cc:2454] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 0.
I20260925 10:25:23.914659 15266 leader_election.cc:302] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.914961 15554 raft_consensus.cc:2781] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:23.915069 15554 raft_consensus.cc:482] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:23.915233 15554 raft_consensus.cc:3037] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.917865 15554 raft_consensus.cc:504] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.918491 15554 leader_election.cc:288] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 election: Requested vote from peers ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:23.919057 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087f851497a5471983c7620615e499b5" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae36637e147c4564956090cf8697dad1"
I20260925 10:25:23.919054 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087f851497a5471983c7620615e499b5" candidate_uuid: "74a6c5c2a0254d14a04896cb663183a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
I20260925 10:25:23.919270 15479 raft_consensus.cc:3037] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.919281 15196 raft_consensus.cc:3037] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:23.922216 15196 raft_consensus.cc:2454] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 1.
I20260925 10:25:23.922216 15479 raft_consensus.cc:2454] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74a6c5c2a0254d14a04896cb663183a3 in term 1.
I20260925 10:25:23.922636 15266 leader_election.cc:302] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:23.922915 15554 raft_consensus.cc:2781] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:23.923101 15554 raft_consensus.cc:686] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 LEADER]: Becoming Leader. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Running, Role: LEADER
I20260925 10:25:23.923389 15554 consensus_queue.cc:237] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:23.926628 15039 catalog_manager.cc:5495] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "74a6c5c2a0254d14a04896cb663183a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } health_report { overall_health: UNKNOWN } } }
I20260925 10:25:24.005134 15196 raft_consensus.cc:1261] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Refusing update from remote peer 74a6c5c2a0254d14a04896cb663183a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:24.005345 15479 raft_consensus.cc:1261] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Refusing update from remote peer 74a6c5c2a0254d14a04896cb663183a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:24.006062 15554 consensus_queue.cc:1035] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:24.006516 15554 consensus_queue.cc:1035] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:24.023334 15572 mvcc.cc:205] Tried to move back new op lower bound from 7333199560720084992 to 7333199560392101888. Current Snapshot: MvccSnapshot[applied={T|T < 7333199560720084992}]
W20260925 10:25:24.037788 15573 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:25:25.630874 15039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:25.631120 15039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:25.635617 15039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Sending DeleteTablet for 3 replicas of tablet 087f851497a5471983c7620615e499b5
I20260925 10:25:25.636516 15315 tablet_service.cc:1514] Processing DeleteTablet for tablet 087f851497a5471983c7620615e499b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56346
I20260925 10:25:25.636584 15176 tablet_service.cc:1514] Processing DeleteTablet for tablet 087f851497a5471983c7620615e499b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41738
I20260925 10:25:25.636590 15453 tablet_service.cc:1514] Processing DeleteTablet for tablet 087f851497a5471983c7620615e499b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:38350
I20260925 10:25:25.649122 15628 tablet_replica.cc:330] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: stopping tablet replica
I20260925 10:25:25.649331 15626 tablet_replica.cc:330] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: stopping tablet replica
I20260925 10:25:25.649430 15628 raft_consensus.cc:2229] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:25.649569 15626 raft_consensus.cc:2229] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:25.649739 15628 raft_consensus.cc:2258] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.649988 15626 raft_consensus.cc:2258] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.651443 15626 ts_tablet_manager.cc:1905] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.654209 15628 ts_tablet_manager.cc:1905] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.657925 15627 tablet_replica.cc:330] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: stopping tablet replica
I20260925 10:25:25.658183 15627 raft_consensus.cc:2229] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:25.658473 15627 raft_consensus.cc:2258] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.659579 15627 ts_tablet_manager.cc:1905] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.659816 15626 ts_tablet_manager.cc:1918] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:25.660015 15626 log.cc:1196] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/087f851497a5471983c7620615e499b5
I20260925 10:25:25.660516 15626 ts_tablet_manager.cc:1939] T 087f851497a5471983c7620615e499b5 P 74a6c5c2a0254d14a04896cb663183a3: Deleting consensus metadata
I20260925 10:25:25.661808 15025 catalog_manager.cc:4841] TS 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773): tablet 087f851497a5471983c7620615e499b5 (table test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]) successfully deleted
I20260925 10:25:25.663460 15628 ts_tablet_manager.cc:1918] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:25.663609 15628 log.cc:1196] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/087f851497a5471983c7620615e499b5
I20260925 10:25:25.664152 15628 ts_tablet_manager.cc:1939] T 087f851497a5471983c7620615e499b5 P ae36637e147c4564956090cf8697dad1: Deleting consensus metadata
I20260925 10:25:25.666025 15627 ts_tablet_manager.cc:1918] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:25.666193 15627 log.cc:1196] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/087f851497a5471983c7620615e499b5
I20260925 10:25:25.666195 15023 catalog_manager.cc:4841] TS ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859): tablet 087f851497a5471983c7620615e499b5 (table test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]) successfully deleted
I20260925 10:25:25.666620 15627 ts_tablet_manager.cc:1939] T 087f851497a5471983c7620615e499b5 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting consensus metadata
I20260925 10:25:25.667623 15023 catalog_manager.cc:4841] TS ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127): tablet 087f851497a5471983c7620615e499b5 (table test-table [id=3e28b99bdb6f4212a14c748b7d13ac7c]) successfully deleted
Sep 25 10:25:25 dist-test-slave-f9rl krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:25:25 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:25.699297 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.683526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.12.248.126:34671)
0925 10:25:25.684096 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.684101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.684512 (+ 411us) server_negotiation.cc:407] Connection header received
0925 10:25:25.684670 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.684674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.684748 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.684850 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:25.686513 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.687390 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.688682 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.689073 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.689206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.693846 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:25.693851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:25.693855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:25.693892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:25.696730 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:25.697551 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:25.697555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:25.697557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:25.697672 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:25.698038 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:25.698040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:25.698042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:25.698318 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:25.698503 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.698773 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.698933 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":320,"threads_started":1}
I20260925 10:25:25.703496 15039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:25.704059 15039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:25.710726 15176 tablet_service.cc:1467] Processing CreateTablet for tablet 6739ff0746e54ae5b797d48500d472c8 (DEFAULT_TABLE table=test-table [id=0394521885054be9bbfea112f7ab9c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:25.710906 15315 tablet_service.cc:1467] Processing CreateTablet for tablet 6739ff0746e54ae5b797d48500d472c8 (DEFAULT_TABLE table=test-table [id=0394521885054be9bbfea112f7ab9c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:25.711274 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739ff0746e54ae5b797d48500d472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:25.711354 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739ff0746e54ae5b797d48500d472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:25.711529 15453 tablet_service.cc:1467] Processing CreateTablet for tablet 6739ff0746e54ae5b797d48500d472c8 (DEFAULT_TABLE table=test-table [id=0394521885054be9bbfea112f7ab9c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:25.712117 15453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739ff0746e54ae5b797d48500d472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:25.717959 15637 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Bootstrap starting.
I20260925 10:25:25.718709 15638 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Bootstrap starting.
I20260925 10:25:25.720391 15638 tablet_bootstrap.cc:654] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:25.720391 15637 tablet_bootstrap.cc:654] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:25.722713 15638 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: No bootstrap required, opened a new log
I20260925 10:25:25.722867 15638 ts_tablet_manager.cc:1397] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:25:25.723327 15637 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: No bootstrap required, opened a new log
I20260925 10:25:25.723479 15637 ts_tablet_manager.cc:1397] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260925 10:25:25.723711 15638 raft_consensus.cc:348] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.723946 15638 raft_consensus.cc:374] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:25.724033 15638 raft_consensus.cc:729] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36637e147c4564956090cf8697dad1, State: Initialized, Role: FOLLOWER
I20260925 10:25:25.724238 15637 raft_consensus.cc:348] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.724298 15638 consensus_queue.cc:260] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.724476 15637 raft_consensus.cc:374] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:25.724570 15637 raft_consensus.cc:729] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Initialized, Role: FOLLOWER
I20260925 10:25:25.724869 15637 consensus_queue.cc:260] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.725193 15638 ts_tablet_manager.cc:1428] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:25:25.725322 15639 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Bootstrap starting.
I20260925 10:25:25.725795 15637 ts_tablet_manager.cc:1428] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:25:25.728529 15639 tablet_bootstrap.cc:654] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:25.732556 15639 tablet_bootstrap.cc:492] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: No bootstrap required, opened a new log
I20260925 10:25:25.732820 15639 ts_tablet_manager.cc:1397] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260925 10:25:25.734120 15639 raft_consensus.cc:348] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.734426 15639 raft_consensus.cc:374] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:25.734647 15639 raft_consensus.cc:729] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Initialized, Role: FOLLOWER
I20260925 10:25:25.734979 15639 consensus_queue.cc:260] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.735807 15639 ts_tablet_manager.cc:1428] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:25:25.836498 15642 raft_consensus.cc:482] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:25.836792 15642 raft_consensus.cc:504] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.838446 15642 leader_election.cc:288] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:25.850589 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.838980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:32773 (local address 127.12.248.65:45823)
0925 10:25:25.841346 (+ 2366us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:25.841372 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:25.841654 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:25.842364 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:25.842372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:25.842406 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:25.842851 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.842867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.844895 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.844904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.848195 (+ 3291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.848218 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.849736 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.849744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.849936 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.849960 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:25:25.850095 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:25:25.850254 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2185,"thread_start_us":9620,"threads_started":1}
I20260925 10:25:25.850589 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.839460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:45823 (local address 127.12.248.66:32773)
0925 10:25:25.840790 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.840800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.841506 (+ 706us) server_negotiation.cc:407] Connection header received
0925 10:25:25.841873 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.841881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.841988 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.842096 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:25.843111 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.844656 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.848446 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.849417 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.849682 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.849774 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.850114 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.850253 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":961,"threads_started":1}
I20260925 10:25:25.851332 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739ff0746e54ae5b797d48500d472c8" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3" is_pre_election: true
I20260925 10:25:25.851647 15340 raft_consensus.cc:2454] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 0.
I20260925 10:25:25.852173 15128 leader_election.cc:302] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ae36637e147c4564956090cf8697dad1; no voters:
I20260925 10:25:25.852649 15642 raft_consensus.cc:2781] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:25.852820 15642 raft_consensus.cc:482] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:25.852964 15642 raft_consensus.cc:3037] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:25.855858 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.839117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.12.248.65:42233)
0925 10:25:25.848913 (+ 9796us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:25.848929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:25.849091 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:25.849419 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:25.849423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:25.849480 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:25.850205 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.850215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.851556 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.851564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.853700 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.853716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.855289 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.855293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.855412 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.855435 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:25:25.855524 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:25:25.855611 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9683,"thread_start_us":1841,"threads_started":1}
I20260925 10:25:25.856259 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.838917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:42233 (local address 127.12.248.67:38127)
0925 10:25:25.848691 (+ 9774us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.848697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.849025 (+ 328us) server_negotiation.cc:407] Connection header received
0925 10:25:25.849228 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.849232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.849296 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.849402 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:25.850382 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.851380 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.853859 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.855074 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.855259 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.855335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.855593 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.855729 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9609,"thread_start_us":9433,"threads_started":1}
I20260925 10:25:25.856230 15642 raft_consensus.cc:504] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.857007 15642 leader_election.cc:288] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 election: Requested vote from peers 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:25.856909 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739ff0746e54ae5b797d48500d472c8" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" is_pre_election: true
I20260925 10:25:25.857183 15479 raft_consensus.cc:2454] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 0.
I20260925 10:25:25.857396 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739ff0746e54ae5b797d48500d472c8" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3"
I20260925 10:25:25.857672 15340 raft_consensus.cc:3037] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:25.857674 15478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739ff0746e54ae5b797d48500d472c8" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
I20260925 10:25:25.857882 15478 raft_consensus.cc:3037] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:25.859856 15340 raft_consensus.cc:2454] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 1.
I20260925 10:25:25.860082 15478 raft_consensus.cc:2454] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 1.
I20260925 10:25:25.860267 15128 leader_election.cc:302] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ae36637e147c4564956090cf8697dad1; no voters:
I20260925 10:25:25.860589 15642 raft_consensus.cc:2781] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:25.861063 15642 raft_consensus.cc:686] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 1 LEADER]: Becoming Leader. State: Replica: ae36637e147c4564956090cf8697dad1, State: Running, Role: LEADER
I20260925 10:25:25.861466 15642 consensus_queue.cc:237] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:25.865491 15038 catalog_manager.cc:5495] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae36637e147c4564956090cf8697dad1 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "ae36637e147c4564956090cf8697dad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } health_report { overall_health: UNKNOWN } } }
I20260925 10:25:25.892582 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.882181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.12.248.65:46859)
0925 10:25:25.885870 (+ 3689us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.885874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.885899 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:25:25.885986 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.885990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.886074 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.886220 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:25.886594 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.887706 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.889122 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.889785 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.889939 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.890642 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:25.891518 (+ 876us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:25.891740 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.891913 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.892047 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3559,"thread_start_us":3431,"threads_started":1}
I20260925 10:25:25.909145 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.903614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:52307 (local address 127.12.248.67:38127)
0925 10:25:25.903769 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.903774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.904507 (+ 733us) server_negotiation.cc:407] Connection header received
0925 10:25:25.904735 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.904739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.904802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.904899 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:25.905417 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.906153 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.907498 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.907896 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.908034 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.908453 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:25.908546 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:25.908640 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.908777 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.908850 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:25:25.909255 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.903707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38127 (local address 127.12.248.65:52307)
0925 10:25:25.904407 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:25.904422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:25.904584 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:25.904915 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:25.904919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:25.904931 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:25.905280 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.905289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.906298 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.906303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.907350 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.907360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.908063 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.908070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.908222 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.908271 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:25.908641 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:25.908654 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:25:25.908757 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:25:25.908885 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":422,"threads_started":1}
I20260925 10:25:25.917909 15478 raft_consensus.cc:1261] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Refusing update from remote peer ae36637e147c4564956090cf8697dad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:25.917938 15340 raft_consensus.cc:1261] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Refusing update from remote peer ae36637e147c4564956090cf8697dad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:25.918568 15648 consensus_queue.cc:1035] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:25.918953 15642 consensus_queue.cc:1035] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:25.946576 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.940649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37037 (local address 127.12.248.65:46859)
0925 10:25:25.940814 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:25.940819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:25.941188 (+ 369us) server_negotiation.cc:407] Connection header received
0925 10:25:25.941402 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:25.941406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:25.941523 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:25.941605 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:25.942073 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.942870 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.944949 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.945392 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.945555 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.945938 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:25.946028 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:25.946094 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:25.946300 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 10:25:25.946359 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:25:25.946828 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:25.940335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:46859 (local address 127.12.248.67:37037)
0925 10:25:25.941091 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:25.941114 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:25.941275 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:25.941650 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:25.941653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:25.941663 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:25:25.941947 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.941955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.943022 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.943029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.944808 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:25.944823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:25.945604 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:25.945609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:25.945738 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:25.945794 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:25:25.946173 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:25:25.946184 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:25:25.946283 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:25:25.946386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":603,"threads_started":1}
I20260925 10:25:25.982851 15039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:25.983158 15039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:25.986656 15039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Sending DeleteTablet for 3 replicas of tablet 6739ff0746e54ae5b797d48500d472c8
I20260925 10:25:25.987722 15453 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739ff0746e54ae5b797d48500d472c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:38350
I20260925 10:25:25.988049 15627 tablet_replica.cc:330] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: stopping tablet replica
I20260925 10:25:25.988152 15315 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739ff0746e54ae5b797d48500d472c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56346
I20260925 10:25:25.988317 15627 raft_consensus.cc:2229] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:25.988374 15176 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739ff0746e54ae5b797d48500d472c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:25 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41738
I20260925 10:25:25.988528 15627 raft_consensus.cc:2258] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.988603 15626 tablet_replica.cc:330] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: stopping tablet replica
I20260925 10:25:25.988651 15628 tablet_replica.cc:330] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: stopping tablet replica
I20260925 10:25:25.988829 15628 raft_consensus.cc:2229] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:25.988905 15626 raft_consensus.cc:2229] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:25.989178 15626 raft_consensus.cc:2258] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.989297 15628 raft_consensus.cc:2258] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:25.989930 15627 ts_tablet_manager.cc:1905] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.990371 15626 ts_tablet_manager.cc:1905] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.990852 15039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:25.991649 15039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:25.993351 15628 ts_tablet_manager.cc:1905] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:25.998672 15626 ts_tablet_manager.cc:1918] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:25.998828 15626 log.cc:1196] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/6739ff0746e54ae5b797d48500d472c8
I20260925 10:25:25.999331 15626 ts_tablet_manager.cc:1939] T 6739ff0746e54ae5b797d48500d472c8 P 74a6c5c2a0254d14a04896cb663183a3: Deleting consensus metadata
I20260925 10:25:26.000557 15025 catalog_manager.cc:4841] TS 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773): tablet 6739ff0746e54ae5b797d48500d472c8 (table test-table [id=0394521885054be9bbfea112f7ab9c3b]) successfully deleted
I20260925 10:25:26.003119 15176 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a (DEFAULT_TABLE table=test-table [id=850dee9330bc44f3ad47004ad89df44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:26.003309 15628 ts_tablet_manager.cc:1918] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:26.003456 15628 log.cc:1196] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/6739ff0746e54ae5b797d48500d472c8
I20260925 10:25:26.003784 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6617be59f43f1b8ba0a20d56e7c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:26.004199 15628 ts_tablet_manager.cc:1939] T 6739ff0746e54ae5b797d48500d472c8 P ae36637e147c4564956090cf8697dad1: Deleting consensus metadata
I20260925 10:25:26.005421 15023 catalog_manager.cc:4841] TS ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859): tablet 6739ff0746e54ae5b797d48500d472c8 (table test-table [id=0394521885054be9bbfea112f7ab9c3b]) successfully deleted
I20260925 10:25:26.005522 15627 ts_tablet_manager.cc:1918] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:25:26.005781 15453 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a (DEFAULT_TABLE table=test-table [id=850dee9330bc44f3ad47004ad89df44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:26.005874 15627 log.cc:1196] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/6739ff0746e54ae5b797d48500d472c8
I20260925 10:25:26.006369 15453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6617be59f43f1b8ba0a20d56e7c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:26.006748 15627 ts_tablet_manager.cc:1939] T 6739ff0746e54ae5b797d48500d472c8 P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting consensus metadata
I20260925 10:25:26.006698 15315 tablet_service.cc:1467] Processing CreateTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a (DEFAULT_TABLE table=test-table [id=850dee9330bc44f3ad47004ad89df44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:26.007265 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b6617be59f43f1b8ba0a20d56e7c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:26.008311 15023 catalog_manager.cc:4841] TS ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127): tablet 6739ff0746e54ae5b797d48500d472c8 (table test-table [id=0394521885054be9bbfea112f7ab9c3b]) successfully deleted
I20260925 10:25:26.014127 15638 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Bootstrap starting.
I20260925 10:25:26.014541 15639 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Bootstrap starting.
I20260925 10:25:26.016515 15638 tablet_bootstrap.cc:654] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:26.016654 15639 tablet_bootstrap.cc:654] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:26.017455 15637 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Bootstrap starting.
I20260925 10:25:26.019860 15637 tablet_bootstrap.cc:654] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:26.026659 15637 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: No bootstrap required, opened a new log
I20260925 10:25:26.026947 15637 ts_tablet_manager.cc:1397] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 10:25:26.027175 15638 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: No bootstrap required, opened a new log
I20260925 10:25:26.027383 15638 ts_tablet_manager.cc:1397] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260925 10:25:26.028177 15637 raft_consensus.cc:348] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.028182 15638 raft_consensus.cc:348] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.028487 15638 raft_consensus.cc:374] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:26.028613 15638 raft_consensus.cc:729] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36637e147c4564956090cf8697dad1, State: Initialized, Role: FOLLOWER
I20260925 10:25:26.028620 15637 raft_consensus.cc:374] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:26.028829 15637 raft_consensus.cc:729] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a6c5c2a0254d14a04896cb663183a3, State: Initialized, Role: FOLLOWER
I20260925 10:25:26.028957 15638 consensus_queue.cc:260] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.029258 15637 consensus_queue.cc:260] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.029677 15638 ts_tablet_manager.cc:1428] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 10:25:26.031185 15637 ts_tablet_manager.cc:1428] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:25:26.032202 15639 tablet_bootstrap.cc:492] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: No bootstrap required, opened a new log
I20260925 10:25:26.032397 15639 ts_tablet_manager.cc:1397] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.007s
I20260925 10:25:26.033416 15639 raft_consensus.cc:348] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.033704 15639 raft_consensus.cc:374] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:26.033787 15639 raft_consensus.cc:729] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee44c88e6c1b4d6ebfca31df2125c6c4, State: Initialized, Role: FOLLOWER
I20260925 10:25:26.034027 15639 consensus_queue.cc:260] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.034627 15639 ts_tablet_manager.cc:1428] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:25:26.116453 15642 raft_consensus.cc:482] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:26.116698 15642 raft_consensus.cc:504] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.118000 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6617be59f43f1b8ba0a20d56e7c6a" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3" is_pre_election: true
I20260925 10:25:26.118108 15478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6617be59f43f1b8ba0a20d56e7c6a" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" is_pre_election: true
I20260925 10:25:26.118263 15340 raft_consensus.cc:2454] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 0.
I20260925 10:25:26.118394 15478 raft_consensus.cc:2454] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 0.
I20260925 10:25:26.118803 15128 leader_election.cc:302] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74a6c5c2a0254d14a04896cb663183a3, ae36637e147c4564956090cf8697dad1; no voters:
I20260925 10:25:26.119072 15642 leader_election.cc:288] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:26.119144 15660 raft_consensus.cc:2781] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:26.119297 15660 raft_consensus.cc:482] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:26.119414 15660 raft_consensus.cc:3037] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:26.122846 15660 raft_consensus.cc:504] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.123695 15660 leader_election.cc:288] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 election: Requested vote from peers 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773), ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127)
I20260925 10:25:26.124136 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6617be59f43f1b8ba0a20d56e7c6a" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74a6c5c2a0254d14a04896cb663183a3"
I20260925 10:25:26.124154 15478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b6617be59f43f1b8ba0a20d56e7c6a" candidate_uuid: "ae36637e147c4564956090cf8697dad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4"
I20260925 10:25:26.124358 15340 raft_consensus.cc:3037] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:26.124385 15478 raft_consensus.cc:3037] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:26.126657 15478 raft_consensus.cc:2454] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 1.
I20260925 10:25:26.127175 15126 leader_election.cc:302] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae36637e147c4564956090cf8697dad1, ee44c88e6c1b4d6ebfca31df2125c6c4; no voters:
I20260925 10:25:26.127477 15340 raft_consensus.cc:2454] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae36637e147c4564956090cf8697dad1 in term 1.
I20260925 10:25:26.127802 15660 raft_consensus.cc:2781] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:26.127976 15660 raft_consensus.cc:686] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 1 LEADER]: Becoming Leader. State: Replica: ae36637e147c4564956090cf8697dad1, State: Running, Role: LEADER
I20260925 10:25:26.128240 15660 consensus_queue.cc:237] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } }
I20260925 10:25:26.131533 15037 catalog_manager.cc:5495] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae36637e147c4564956090cf8697dad1 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "ae36637e147c4564956090cf8697dad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae36637e147c4564956090cf8697dad1" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 } health_report { overall_health: UNKNOWN } } }
I20260925 10:25:26.161291 15478 raft_consensus.cc:1261] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Refusing update from remote peer ae36637e147c4564956090cf8697dad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:26.161291 15336 raft_consensus.cc:1261] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Refusing update from remote peer ae36637e147c4564956090cf8697dad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:26.161991 15642 consensus_queue.cc:1035] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74a6c5c2a0254d14a04896cb663183a3" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:26.162313 15660 consensus_queue.cc:1035] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee44c88e6c1b4d6ebfca31df2125c6c4" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:25:26.203279 15609 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:25:27.793514 15039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:27.793769 15039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:27.798136 15039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bf4f142c9894edf87ca529232549f99: Sending DeleteTablet for 3 replicas of tablet 10b6617be59f43f1b8ba0a20d56e7c6a
I20260925 10:25:27.799027 15315 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:27 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56346
I20260925 10:25:27.799048 15176 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:27 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41738
I20260925 10:25:27.799072 15453 tablet_service.cc:1514] Processing DeleteTablet for tablet 10b6617be59f43f1b8ba0a20d56e7c6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:27 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:38350
I20260925 10:25:27.800498 15693 tablet_replica.cc:330] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: stopping tablet replica
I20260925 10:25:27.800763 15693 raft_consensus.cc:2229] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:27.800834 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15110
I20260925 10:25:27.801141 15693 raft_consensus.cc:2258] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:27.801806 15691 tablet_replica.cc:330] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: stopping tablet replica
I20260925 10:25:27.802073 15691 raft_consensus.cc:2229] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:27.802456 15691 raft_consensus.cc:2258] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:27.804241 15691 ts_tablet_manager.cc:1905] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:27.805078 15693 ts_tablet_manager.cc:1905] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:27.810407 15692 tablet_replica.cc:330] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: stopping tablet replica
I20260925 10:25:27.810684 15692 raft_consensus.cc:2229] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:27.810968 15692 raft_consensus.cc:2258] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:27.812577 15692 ts_tablet_manager.cc:1905] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:27.812623 15693 ts_tablet_manager.cc:1918] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:27.812783 15693 log.cc:1196] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/10b6617be59f43f1b8ba0a20d56e7c6a
I20260925 10:25:27.813218 15693 ts_tablet_manager.cc:1939] T 10b6617be59f43f1b8ba0a20d56e7c6a P ae36637e147c4564956090cf8697dad1: Deleting consensus metadata
I20260925 10:25:27.814543 15023 catalog_manager.cc:4841] TS ae36637e147c4564956090cf8697dad1 (127.12.248.65:46859): tablet 10b6617be59f43f1b8ba0a20d56e7c6a (table test-table [id=850dee9330bc44f3ad47004ad89df44a]) successfully deleted
I20260925 10:25:27.817248 15691 ts_tablet_manager.cc:1918] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:27.817462 15691 log.cc:1196] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/10b6617be59f43f1b8ba0a20d56e7c6a
I20260925 10:25:27.818015 15691 ts_tablet_manager.cc:1939] T 10b6617be59f43f1b8ba0a20d56e7c6a P 74a6c5c2a0254d14a04896cb663183a3: Deleting consensus metadata
I20260925 10:25:27.818608 15692 ts_tablet_manager.cc:1918] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:25:27.818776 15692 log.cc:1196] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/10b6617be59f43f1b8ba0a20d56e7c6a
I20260925 10:25:27.819280 15692 ts_tablet_manager.cc:1939] T 10b6617be59f43f1b8ba0a20d56e7c6a P ee44c88e6c1b4d6ebfca31df2125c6c4: Deleting consensus metadata
I20260925 10:25:27.819273 15025 catalog_manager.cc:4841] TS 74a6c5c2a0254d14a04896cb663183a3 (127.12.248.66:32773): tablet 10b6617be59f43f1b8ba0a20d56e7c6a (table test-table [id=850dee9330bc44f3ad47004ad89df44a]) successfully deleted
I20260925 10:25:27.820371 15023 catalog_manager.cc:4841] TS ee44c88e6c1b4d6ebfca31df2125c6c4 (127.12.248.67:38127): tablet 10b6617be59f43f1b8ba0a20d56e7c6a (table test-table [id=850dee9330bc44f3ad47004ad89df44a]) successfully deleted
Sep 25 10:25:27 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:27.841073 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:27.802074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.12.248.65:46859)
0925 10:25:27.802771 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:27.802781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:27.811002 (+ 8221us) server_negotiation.cc:407] Connection header received
0925 10:25:27.811279 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:27.811286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:27.811418 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:27.811551 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:27.813545 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.814777 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.816654 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.817577 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.817717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:27.835456 (+ 17739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:27.835474 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:27.835491 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:27.835532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:27.838585 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.839315 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.839322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.839324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.839453 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.839803 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.839807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.839809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.840140 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:27.840350 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:27.840607 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:25:27.840804 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":297,"threads_started":1}
W20260925 10:25:27.841857 15241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940
W20260925 10:25:27.842463 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:27.842684 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15110
I20260925 10:25:27.865108 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15250
Sep 25 10:25:27 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:27.881212 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:27.865952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.12.248.66:32773)
0925 10:25:27.866484 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:27.866488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:27.866505 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:25:27.866556 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:27.866560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:27.866621 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:27.866760 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:27.868192 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.868918 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.870372 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.870865 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.870984 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:27.875336 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:27.875351 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:27.875369 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:27.875413 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:27.878680 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.879346 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.879354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.879356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.879489 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.879829 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.879833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.879835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.880231 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:27.880466 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:27.880778 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 10:25:27.880950 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":293,"threads_started":1}
W20260925 10:25:27.882004 15380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34876
W20260925 10:25:27.882681 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:27.882867 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15250
I20260925 10:25:27.906896 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15389
Sep 25 10:25:27 dist-test-slave-f9rl krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:27.922252 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:27.907655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.12.248.67:38127)
0925 10:25:27.908186 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:27.908190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:27.908206 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:25:27.908261 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:27.908264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:27.908326 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:27.908416 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:27.909817 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.910529 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.911755 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:27.912138 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:27.912226 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:27.916625 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:27.916639 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:27.916656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:27.916693 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:27.919451 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.920122 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.920129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.920132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.920259 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:27.920632 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:27.920638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:27.920641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:27.921012 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:27.921315 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:27.921661 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:25:27.921896 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
W20260925 10:25:27.922899 15519 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41840
W20260925 10:25:27.923512 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:27.923691 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15389
I20260925 10:25:27.945015 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15007
I20260925 10:25:27.946272 15079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:28.065521 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15007
2026-09-25T10:25:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8510 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:28 dist-test-slave-f9rl krb5kdc[15701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:28 dist-test-slave-f9rl krb5kdc[15701](info): set up 2 sockets
Sep 25 10:25:28 dist-test-slave-f9rl krb5kdc[15701](info): commencing operation
krb5kdc: starting...
W20260925 10:25:29.171310 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:29 dist-test-slave-f9rl krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:29.365950 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37047
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:29.688973 15714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:29.689325 15714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:29.689550 15714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:29.696732 15714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:29.696894 15714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:29.697005 15714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:29.697117 15714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:29.697206 15714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:29.707072 15714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:29.708406 15714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:29.709851 15714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:29.722242 15719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:29.722537 15714 server_base.cc:1034] running on GCE node
W20260925 10:25:29.722165 15722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:29.722157 15720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:29.723639 15714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:29.725051 15714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:29.726249 15714 hybrid_clock.cc:648] HybridClock initialized: now 1790331929726190 us; error 61 us; skew 500 ppm
I20260925 10:25:29.726593 15714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:29 dist-test-slave-f9rl krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331929, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:29.729676 15714 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:29.731076 15714 webserver.cc:460] Webserver started at http://127.12.248.126:43149/ using document root <none> and password file <none>
I20260925 10:25:29.731750 15714 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:29.731895 15714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:29.732199 15714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:29.735296 15714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "27561da902ec4c3e896a28bae7a36705"
format_stamp: "Formatted at 2026-09-25 10:25:29 on dist-test-slave-f9rl"
server_key: "8f9e59225edd1002c637467557ed4fe1"
server_key_iv: "2c877bb716338da686b718a1a1e24810"
server_key_version: "encryptionkey@0"
I20260925 10:25:29.736074 15714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "27561da902ec4c3e896a28bae7a36705"
format_stamp: "Formatted at 2026-09-25 10:25:29 on dist-test-slave-f9rl"
server_key: "8f9e59225edd1002c637467557ed4fe1"
server_key_iv: "2c877bb716338da686b718a1a1e24810"
server_key_version: "encryptionkey@0"
I20260925 10:25:29.741339 15714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:25:29.744974 15729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:29.746650 15714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:29.746852 15714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "27561da902ec4c3e896a28bae7a36705"
format_stamp: "Formatted at 2026-09-25 10:25:29 on dist-test-slave-f9rl"
server_key: "8f9e59225edd1002c637467557ed4fe1"
server_key_iv: "2c877bb716338da686b718a1a1e24810"
server_key_version: "encryptionkey@0"
I20260925 10:25:29.747066 15714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:29.767910 15714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:29.771895 15714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:29.772203 15714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:29.791291 15714 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37047
I20260925 10:25:29.791333 15780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37047 every 8 connection(s)
I20260925 10:25:29.792765 15714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:29.795511 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15714
I20260925 10:25:29.795749 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:29.796546 13281 external_mini_cluster.cc:1428] Setting key a5b4730874f73a28ec1d6c5f7dc765cb
I20260925 10:25:29.797155 15781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:25:29 dist-test-slave-f9rl krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:29.811781 15781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705: Bootstrap starting.
I20260925 10:25:29.815217 15781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:29.816299 15781 log.cc:824] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:29.819336 15781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705: No bootstrap required, opened a new log
I20260925 10:25:29.820153 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:29.798612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.12.248.126:37047)
0925 10:25:29.799376 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:29.799387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:29.799427 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:25:29.801730 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:29.801763 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:29.802140 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:29.802459 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:29.804274 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:29.805600 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:29.807337 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:29.808048 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:29.808181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:29.812767 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:29.812774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:29.812788 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:29.812827 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:29.815915 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:29.816603 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:29.816610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:29.816616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:29.816789 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:29.817201 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:29.817208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:29.817216 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:29.817846 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:29.818148 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:29.818608 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 10:25:29.819043 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":367,"threads_started":1}
I20260925 10:25:29.827044 15781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } }
I20260925 10:25:29.827505 15781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:29.827613 15781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27561da902ec4c3e896a28bae7a36705, State: Initialized, Role: FOLLOWER
I20260925 10:25:29.828222 15781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } }
I20260925 10:25:29.828522 15781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:29.828637 15781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:29.828759 15781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:29.830952 15781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } }
I20260925 10:25:29.831375 15781 leader_election.cc:302] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27561da902ec4c3e896a28bae7a36705; no voters:
I20260925 10:25:29.831997 15781 leader_election.cc:288] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:29.832125 15786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:29.833083 15786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [term 1 LEADER]: Becoming Leader. State: Replica: 27561da902ec4c3e896a28bae7a36705, State: Running, Role: LEADER
I20260925 10:25:29.833643 15786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } }
I20260925 10:25:29.834362 15781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:29.836525 15787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27561da902ec4c3e896a28bae7a36705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } } }
I20260925 10:25:29.836884 15787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:29.836535 15788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27561da902ec4c3e896a28bae7a36705. Latest consensus state: current_term: 1 leader_uuid: "27561da902ec4c3e896a28bae7a36705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27561da902ec4c3e896a28bae7a36705" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37047 } } }
I20260925 10:25:29.837057 15788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:29.837777 15793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:29.843222 15793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:29.852913 15793 catalog_manager.cc:1347] Generated new cluster ID: 398f6d3c9a9b4386b443b77a5341f0a3
I20260925 10:25:29.853044 15793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:29.874701 15793 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:29.876704 15793 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:29.892920 15793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27561da902ec4c3e896a28bae7a36705: Generated new TSK 0
I20260925 10:25:29.894219 15793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:29.951704 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:30.317895 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:30.318238 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:30.318396 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:30.325752 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:30.325912 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:30.326090 15807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:30.336371 15807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:30.337863 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:30.339300 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:30.353329 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:30.353314 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:30.353466 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:30.354130 15807 server_base.cc:1034] running on GCE node
I20260925 10:25:30.354689 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:30.355566 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:30.356793 15807 hybrid_clock.cc:648] HybridClock initialized: now 1790331930356746 us; error 30 us; skew 500 ppm
I20260925 10:25:30.357169 15807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:30 dist-test-slave-f9rl krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331930, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:30.360424 15807 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:30.362036 15807 webserver.cc:460] Webserver started at http://127.12.248.65:43811/ using document root <none> and password file <none>
I20260925 10:25:30.362736 15807 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:30.362867 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:30.363185 15807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:30.366192 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "d1b469c021424f0191364bfb07cb931a"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "c83aeab383429cd1c574b0b198a31c99"
server_key_iv: "dbb0385420d2a139715bb25c84bd76a5"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.366988 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "d1b469c021424f0191364bfb07cb931a"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "c83aeab383429cd1c574b0b198a31c99"
server_key_iv: "dbb0385420d2a139715bb25c84bd76a5"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.372140 15807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:30.375659 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.377081 15807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:30.377259 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "d1b469c021424f0191364bfb07cb931a"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "c83aeab383429cd1c574b0b198a31c99"
server_key_iv: "dbb0385420d2a139715bb25c84bd76a5"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.377485 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:30.393653 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:30.397717 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:30.398002 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:30.399001 15807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:30.400328 15807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:30.400431 15807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.400574 15807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:30.400645 15807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.434540 15807 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:38125
I20260925 10:25:30.434594 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:38125 every 8 connection(s)
I20260925 10:25:30.435956 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:30.444677 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15807
I20260925 10:25:30.444912 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:30.445588 13281 external_mini_cluster.cc:1428] Setting key e210c099a968b6fbef5e9a9bb28936b3
Sep 25 10:25:30 dist-test-slave-f9rl krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331930, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:30.457161 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:30.438520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:34305 (local address 127.12.248.126:37047)
0925 10:25:30.438981 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:30.438985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:30.439959 (+ 974us) server_negotiation.cc:407] Connection header received
0925 10:25:30.441895 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:30.441901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:30.441999 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:30.442115 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:30.443603 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.444329 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.446132 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.446707 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.446846 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:30.450574 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:30.450582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:30.450591 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:30.450657 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:30.453805 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:30.454589 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:30.454597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:30.454604 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:30.454767 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:30.455174 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:30.455181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:30.455187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:30.455565 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:30.455715 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:30.456403 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 10:25:30.456659 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260925 10:25:30.457794 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:30.439015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37047 (local address 127.12.248.65:34305)
0925 10:25:30.439782 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:30.439825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:30.441615 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:30.442354 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:30.442362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:30.442866 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:30.443426 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:30.443443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.444494 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.444506 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:30.445985 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:30.445998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.446940 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.446949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:30.447238 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:30.447929 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:30.447953 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:30.450350 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:30.453947 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:30.453959 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:30.453979 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:30.454454 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:30.454872 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:30.454875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:30.454877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:30.455050 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:30.455682 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:30.455688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:30.456006 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 10:25:30.456221 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 10:25:30.456631 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":266,"threads_started":1}
I20260925 10:25:30.459368 15935 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37047
I20260925 10:25:30.459693 15935 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:30.460677 15935 heartbeater.cc:511] Master 127.12.248.126:37047 requested a full tablet report, sending...
I20260925 10:25:30.463013 15746 ts_manager.cc:194] Registered new tserver with Master: d1b469c021424f0191364bfb07cb931a (127.12.248.65:38125)
I20260925 10:25:30.464424 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:34305
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:30.493177 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:30.815580 15941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:30.815915 15941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:30.816088 15941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:30.823531 15941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:30.823698 15941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:30.823879 15941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:30.833983 15941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:30.835544 15941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:30.837019 15941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:30.850951 15949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:30.851094 15947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:30.851466 15946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:30.851804 15941 server_base.cc:1034] running on GCE node
I20260925 10:25:30.852434 15941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:30.853565 15941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:30.854815 15941 hybrid_clock.cc:648] HybridClock initialized: now 1790331930854765 us; error 42 us; skew 500 ppm
I20260925 10:25:30.855154 15941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:30 dist-test-slave-f9rl krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331930, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:30.858294 15941 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:30.859861 15941 webserver.cc:460] Webserver started at http://127.12.248.66:39229/ using document root <none> and password file <none>
I20260925 10:25:30.860560 15941 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:30.860661 15941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:30.860944 15941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:30.864001 15941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "509b13967df84099808ed1ba923ac0e9"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "439bd3c0dda6cbf891a413590da01122"
server_key_iv: "bcd9cceca105165940caeb1da5a23950"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.864768 15941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "509b13967df84099808ed1ba923ac0e9"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "439bd3c0dda6cbf891a413590da01122"
server_key_iv: "bcd9cceca105165940caeb1da5a23950"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.870507 15941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 10:25:30.874431 15956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.876029 15941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:25:30.876199 15941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "509b13967df84099808ed1ba923ac0e9"
format_stamp: "Formatted at 2026-09-25 10:25:30 on dist-test-slave-f9rl"
server_key: "439bd3c0dda6cbf891a413590da01122"
server_key_iv: "bcd9cceca105165940caeb1da5a23950"
server_key_version: "encryptionkey@0"
I20260925 10:25:30.876435 15941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:30.894997 15941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:30.899365 15941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:30.899673 15941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:30.900671 15941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:30.902201 15941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:30.902274 15941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.902457 15941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:30.902513 15941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:30.937278 15941 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:45053
I20260925 10:25:30.937352 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:45053 every 8 connection(s)
I20260925 10:25:30.938638 15941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:30.942091 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 15941
I20260925 10:25:30.942324 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:30.942945 13281 external_mini_cluster.cc:1428] Setting key 69b1f9eaf78ce1d2bb8e3973278a3b08
Sep 25 10:25:30 dist-test-slave-f9rl krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331930, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:30.962790 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:30.941263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:50303 (local address 127.12.248.126:37047)
0925 10:25:30.941481 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:30.941488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:30.942918 (+ 1430us) server_negotiation.cc:407] Connection header received
0925 10:25:30.945272 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:30.945276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:30.945341 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:30.945425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:30.947661 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.948536 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.951173 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.952000 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.952157 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:30.956194 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:30.956199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:30.956202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:30.956238 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:30.959326 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:30.960105 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:30.960114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:30.960121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:30.960291 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:30.960734 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:30.960741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:30.960746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:30.961127 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:30.961293 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:30.962073 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 10:25:30.962339 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:30.963529 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:30.941812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37047 (local address 127.12.248.66:50303)
0925 10:25:30.942734 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:30.942796 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:30.944933 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:30.945749 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:30.945760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:30.946593 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:30.947476 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:30.947505 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.948714 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.948722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:30.951025 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:30.951039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:30.952313 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:30.952320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:30.952546 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:30.953278 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:30.953306 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:30.955921 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:30.959486 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:30.959494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:30.959510 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:30.959961 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:30.960411 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:30.960414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:30.960416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:30.960594 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:30.961255 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:30.961261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:30.961661 (+ 400us) client_negotiation.cc:769] Sending connection context
0925 10:25:30.961900 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 10:25:30.962322 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":406,"threads_started":1}
I20260925 10:25:30.965050 16069 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37047
I20260925 10:25:30.965327 16069 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:25:30.966130 16069 heartbeater.cc:511] Master 127.12.248.126:37047 requested a full tablet report, sending...
I20260925 10:25:30.967666 15746 ts_manager.cc:194] Registered new tserver with Master: 509b13967df84099808ed1ba923ac0e9 (127.12.248.66:45053)
I20260925 10:25:30.968520 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:50303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:30.992575 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:31.328984 16074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:31.329308 16074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:31.329528 16074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:31.337539 16074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:31.337718 16074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:31.337890 16074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:31.348713 16074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:31.350135 16074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:31.351641 16074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:31.365166 16080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:31.365283 16079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:31.365157 16082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:31.366130 16074 server_base.cc:1034] running on GCE node
I20260925 10:25:31.366828 16074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:31.367800 16074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:31.369036 16074 hybrid_clock.cc:648] HybridClock initialized: now 1790331931368985 us; error 36 us; skew 500 ppm
I20260925 10:25:31.369365 16074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:31 dist-test-slave-f9rl krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331931, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:31.372524 16074 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:31.374054 16074 webserver.cc:460] Webserver started at http://127.12.248.67:39175/ using document root <none> and password file <none>
I20260925 10:25:31.374732 16074 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:31.374833 16074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:31.375088 16074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:31.377997 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "0a7d76c9ae194571893708c7d43cd13c"
format_stamp: "Formatted at 2026-09-25 10:25:31 on dist-test-slave-f9rl"
server_key: "ee64171214143244b0e5cfb1e584c1ae"
server_key_iv: "9b0e267491733612258786f7b506f7c0"
server_key_version: "encryptionkey@0"
I20260925 10:25:31.378769 16074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "0a7d76c9ae194571893708c7d43cd13c"
format_stamp: "Formatted at 2026-09-25 10:25:31 on dist-test-slave-f9rl"
server_key: "ee64171214143244b0e5cfb1e584c1ae"
server_key_iv: "9b0e267491733612258786f7b506f7c0"
server_key_version: "encryptionkey@0"
I20260925 10:25:31.384408 16074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:31.387907 16089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:31.389309 16074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:31.389528 16074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0a7d76c9ae194571893708c7d43cd13c"
format_stamp: "Formatted at 2026-09-25 10:25:31 on dist-test-slave-f9rl"
server_key: "ee64171214143244b0e5cfb1e584c1ae"
server_key_iv: "9b0e267491733612258786f7b506f7c0"
server_key_version: "encryptionkey@0"
I20260925 10:25:31.389731 16074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:31.401163 16074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:31.405210 16074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:31.405551 16074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:31.406513 16074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:31.407853 16074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:31.407927 16074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:31.408075 16074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:31.408128 16074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:31.442382 16074 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:33785
I20260925 10:25:31.442423 16201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:33785 every 8 connection(s)
I20260925 10:25:31.443615 16074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:31.451167 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16074
I20260925 10:25:31.451440 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:31.451884 13281 external_mini_cluster.cc:1428] Setting key c44e3d383e3e186e9acfe59bcfaeeb84
Sep 25 10:25:31 dist-test-slave-f9rl krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331931, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:31.463749 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:31.446307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:59241 (local address 127.12.248.126:37047)
0925 10:25:31.446496 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:31.446501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:31.447582 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 10:25:31.449540 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:31.449544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:31.449628 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:31.449719 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:31.451484 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:31.452322 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:31.454384 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:31.454860 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:31.454989 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:31.458505 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:31.458509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:31.458512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:31.458545 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:31.461040 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:31.461782 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:31.461785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:31.461788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:31.461912 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:31.462288 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:31.462291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:31.462293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:31.462543 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:31.462658 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:31.463327 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 10:25:31.463483 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:31.464752 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:31.446726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37047 (local address 127.12.248.67:59241)
0925 10:25:31.447419 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:31.447463 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:31.449224 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:31.449955 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:31.449963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:31.450622 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:31.451235 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:31.451259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:31.452476 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:31.452481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:31.454238 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:31.454250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:31.455089 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:31.455095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:31.455347 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:31.455969 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:31.455991 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:31.458300 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:31.461166 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:31.461173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:31.461185 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:31.461675 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:31.462006 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:31.462009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:31.462011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:31.462187 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:31.462688 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:31.462694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:31.463004 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 10:25:31.463234 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 10:25:31.463613 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":272,"threads_started":1}
I20260925 10:25:31.466370 16202 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37047
I20260925 10:25:31.466702 16202 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:31.467394 15935 heartbeater.cc:503] Master 127.12.248.126:37047 was elected leader, sending a full tablet report...
I20260925 10:25:31.467538 16202 heartbeater.cc:511] Master 127.12.248.126:37047 requested a full tablet report, sending...
I20260925 10:25:31.468914 15745 ts_manager.cc:194] Registered new tserver with Master: 0a7d76c9ae194571893708c7d43cd13c (127.12.248.67:33785)
I20260925 10:25:31.469782 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:59241
I20260925 10:25:31.479315 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:25:31.509661 15937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:31.507072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.12.248.126:37047)
0925 10:25:31.507328 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:31.507337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:31.507732 (+ 395us) server_negotiation.cc:407] Connection header received
0925 10:25:31.507963 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:31.507967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:31.508032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:31.508141 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:31.509401 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48004: BlockingRecv error: recv got EOF from 127.0.0.1:48004 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:25:31.512930 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15807
W20260925 10:25:31.515852 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15807: Not authorized: Client connection negotiation failed: client connection to 127.12.248.65:38125: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:25:31.516031 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15807
I20260925 10:25:31.529815 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15941
W20260925 10:25:31.532430 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15941: Not authorized: Client connection negotiation failed: client connection to 127.12.248.66:45053: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:25:31.532615 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15941
I20260925 10:25:31.548530 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16074
W20260925 10:25:31.551819 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16074: Not authorized: Client connection negotiation failed: client connection to 127.12.248.67:33785: server requires authentication, but client does not have Kerberos credentials available
I20260925 10:25:31.551949 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16074
I20260925 10:25:31.564095 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 15714
I20260925 10:25:31.565424 15776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:31.669270 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 15714
2026-09-25T10:25:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3600 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:31 dist-test-slave-f9rl krb5kdc[16219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:31 dist-test-slave-f9rl krb5kdc[16219](info): set up 2 sockets
Sep 25 10:25:31 dist-test-slave-f9rl krb5kdc[16219](info): commencing operation
krb5kdc: starting...
W20260925 10:25:32.846870 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 10:25:32.997753 13281 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.151s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 10:25:33.278530 13281 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.281s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 10:25:33.460472 13281 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.182s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/test-user.keytab.
W20260925 10:25:34.223205 13281 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.762s user 0.000s sys 0.004s
Sep 25 10:25:34 dist-test-slave-f9rl krb5kdc[16219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:34.313684 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36187
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:36187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:34.638499 16232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:34.638800 16232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:34.638988 16232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:34.646390 16232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:34.646495 16232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:34.646629 16232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:34.646725 16232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:34.646811 16232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:34.656836 16232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:36187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:34.658326 16232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:34.659775 16232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:34.671692 16237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:34.671630 16240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:34.671729 16238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:34.672676 16232 server_base.cc:1034] running on GCE node
I20260925 10:25:34.673507 16232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:34.674961 16232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:34.676160 16232 hybrid_clock.cc:648] HybridClock initialized: now 1790331934676118 us; error 40 us; skew 500 ppm
I20260925 10:25:34.676482 16232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:34 dist-test-slave-f9rl krb5kdc[16219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331934, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:34.686918 16232 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:34.688484 16232 webserver.cc:460] Webserver started at http://127.12.248.126:39047/ using document root <none> and password file <none>
I20260925 10:25:34.689253 16232 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:34.689357 16232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:34.689687 16232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:34.692652 16232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "957d48942a8f4a33b0fb2ffc308a4164"
format_stamp: "Formatted at 2026-09-25 10:25:34 on dist-test-slave-f9rl"
server_key: "d997742339f9c5dfef1dcaa1ebadd5d6"
server_key_iv: "e164fb429b3a8c9568e5c348fbe0eff2"
server_key_version: "encryptionkey@0"
I20260925 10:25:34.693399 16232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "957d48942a8f4a33b0fb2ffc308a4164"
format_stamp: "Formatted at 2026-09-25 10:25:34 on dist-test-slave-f9rl"
server_key: "d997742339f9c5dfef1dcaa1ebadd5d6"
server_key_iv: "e164fb429b3a8c9568e5c348fbe0eff2"
server_key_version: "encryptionkey@0"
I20260925 10:25:34.698576 16232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:25:34.701906 16247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:34.703240 16232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:25:34.703413 16232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "957d48942a8f4a33b0fb2ffc308a4164"
format_stamp: "Formatted at 2026-09-25 10:25:34 on dist-test-slave-f9rl"
server_key: "d997742339f9c5dfef1dcaa1ebadd5d6"
server_key_iv: "e164fb429b3a8c9568e5c348fbe0eff2"
server_key_version: "encryptionkey@0"
I20260925 10:25:34.703598 16232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:34.723580 16232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:34.728240 16232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:34.728549 16232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:34.747321 16232 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:36187
I20260925 10:25:34.747344 16298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:36187 every 8 connection(s)
I20260925 10:25:34.748730 16232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:34.752490 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16232
I20260925 10:25:34.752724 16299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:34.752759 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:34.753566 13281 external_mini_cluster.cc:1428] Setting key f3bd5e0913d3eff5c537e08bc187fffc
I20260925 10:25:34.768170 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164: Bootstrap starting.
Sep 25 10:25:34 dist-test-slave-f9rl krb5kdc[16219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:34.772356 16299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:34.773792 16299 log.cc:824] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:34.777287 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164: No bootstrap required, opened a new log
I20260925 10:25:34.778455 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:34.755543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.12.248.126:36187)
0925 10:25:34.756364 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:34.756378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:34.756422 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:25:34.758006 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:34.758027 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:34.758415 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:34.758756 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:34.760718 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:34.762082 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:34.763842 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:34.764572 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:34.764711 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:34.769498 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:34.769510 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:34.769529 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:34.769592 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:34.773614 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:34.774498 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:34.774505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:34.774511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:34.774697 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:34.775170 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:34.775174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:34.775177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:34.775892 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:34.776167 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:34.776593 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 10:25:34.777018 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":401,"threads_started":1}
I20260925 10:25:34.782418 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:25:34.785858 16299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } }
I20260925 10:25:34.786265 16299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:34.786370 16299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957d48942a8f4a33b0fb2ffc308a4164, State: Initialized, Role: FOLLOWER
I20260925 10:25:34.786989 16299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } }
I20260925 10:25:34.787354 16299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:34.787494 16299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:34.787671 16299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:34.790503 16299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } }
I20260925 10:25:34.791095 16299 leader_election.cc:302] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 957d48942a8f4a33b0fb2ffc308a4164; no voters:
I20260925 10:25:34.791770 16299 leader_election.cc:288] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:34.791890 16305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:34.793219 16305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [term 1 LEADER]: Becoming Leader. State: Replica: 957d48942a8f4a33b0fb2ffc308a4164, State: Running, Role: LEADER
I20260925 10:25:34.793869 16305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } }
I20260925 10:25:34.794214 16299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:34.796603 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "957d48942a8f4a33b0fb2ffc308a4164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } } }
I20260925 10:25:34.797049 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:34.797475 16307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 957d48942a8f4a33b0fb2ffc308a4164. Latest consensus state: current_term: 1 leader_uuid: "957d48942a8f4a33b0fb2ffc308a4164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "957d48942a8f4a33b0fb2ffc308a4164" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36187 } } }
I20260925 10:25:34.797703 16307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:34.806263 16313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:34.814618 16313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:34.825660 16313 catalog_manager.cc:1347] Generated new cluster ID: dc65004b05884593b7c7dcb0aa414be8
I20260925 10:25:34.825824 16313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:34.834911 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:34.824584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.12.248.126:36187)
0925 10:25:34.824791 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:34.824801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:34.826277 (+ 1476us) server_negotiation.cc:407] Connection header received
0925 10:25:34.826472 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:34.826478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:34.826566 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:34.826688 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:34.827995 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:34.829103 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:34.831013 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:34.831635 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:34.831750 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:34.832995 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:34.833002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:34.833007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:34.834058 (+ 1051us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:34.834127 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:34.834161 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:34.834267 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:34.834435 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:25:34.834527 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:25:34.838024 16313 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:34.839294 16313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:34.855834 16313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 957d48942a8f4a33b0fb2ffc308a4164: Generated new TSK 0
I20260925 10:25:34.856993 16313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:25:34.871747 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16232
I20260925 10:25:34.873052 16294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:34.966046 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16232
2026-09-25T10:25:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3290 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:35 dist-test-slave-f9rl krb5kdc[16333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:35 dist-test-slave-f9rl krb5kdc[16333](info): set up 2 sockets
Sep 25 10:25:35 dist-test-slave-f9rl krb5kdc[16333](info): commencing operation
krb5kdc: starting...
W20260925 10:25:36.075168 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:36 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:36.285580 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44885
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:36.618932 16346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:36.619238 16346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:36.619328 16346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:36.626598 16346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:36.626710 16346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:36.626756 16346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:36.626814 16346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:36.626858 16346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:36.636927 16346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:36.638295 16346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:36.639560 16346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:36.651443 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:36.651510 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:36.652092 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:36.652333 16346 server_base.cc:1034] running on GCE node
I20260925 10:25:36.653095 16346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:36.654608 16346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:36.655839 16346 hybrid_clock.cc:648] HybridClock initialized: now 1790331936655801 us; error 47 us; skew 500 ppm
I20260925 10:25:36.656239 16346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:36 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331936, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:36.659569 16346 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:36.661072 16346 webserver.cc:460] Webserver started at http://127.12.248.126:35033/ using document root <none> and password file <none>
I20260925 10:25:36.661902 16346 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:36.662056 16346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:36.662405 16346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:36.665623 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "23f20e4cd1f446b48d8c75015fd2c4ad"
format_stamp: "Formatted at 2026-09-25 10:25:36 on dist-test-slave-f9rl"
server_key: "03e520c5b90e7a0861bc6f33663886b0"
server_key_iv: "02630787da9f00859d97f191c4b1fe6e"
server_key_version: "encryptionkey@0"
I20260925 10:25:36.666412 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "23f20e4cd1f446b48d8c75015fd2c4ad"
format_stamp: "Formatted at 2026-09-25 10:25:36 on dist-test-slave-f9rl"
server_key: "03e520c5b90e7a0861bc6f33663886b0"
server_key_iv: "02630787da9f00859d97f191c4b1fe6e"
server_key_version: "encryptionkey@0"
I20260925 10:25:36.671597 16346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 10:25:36.675225 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:36.676623 16346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:25:36.676802 16346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "23f20e4cd1f446b48d8c75015fd2c4ad"
format_stamp: "Formatted at 2026-09-25 10:25:36 on dist-test-slave-f9rl"
server_key: "03e520c5b90e7a0861bc6f33663886b0"
server_key_iv: "02630787da9f00859d97f191c4b1fe6e"
server_key_version: "encryptionkey@0"
I20260925 10:25:36.677003 16346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:36.706768 16346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:36.710872 16346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:36.711177 16346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:36.730106 16346 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:44885
I20260925 10:25:36.730140 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:44885 every 8 connection(s)
I20260925 10:25:36.731460 16346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:36.735070 16413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:36.735751 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16346
I20260925 10:25:36.735960 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:36.736577 13281 external_mini_cluster.cc:1428] Setting key 29cf0aef932450224b9645194c12ac9a
I20260925 10:25:36.749130 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad: Bootstrap starting.
Sep 25 10:25:36 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:36.752328 16413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:36.753571 16413 log.cc:824] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:36.756718 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad: No bootstrap required, opened a new log
I20260925 10:25:36.760651 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:36.738630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.12.248.126:44885)
0925 10:25:36.739436 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:36.739453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:36.739513 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:25:36.741634 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:36.741662 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:36.742211 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:36.742657 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:36.744052 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:36.746069 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:36.747407 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:36.748421 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:36.748601 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:36.753016 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:36.753026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:36.753044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:36.753085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:36.756386 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:36.757283 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:36.757289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:36.757294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:36.757515 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:36.757928 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:36.757931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:36.757934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:36.758447 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:36.758674 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:36.759176 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 10:25:36.759570 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":308,"threads_started":1}
I20260925 10:25:36.765728 16413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } }
I20260925 10:25:36.766130 16413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:36.766239 16413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f20e4cd1f446b48d8c75015fd2c4ad, State: Initialized, Role: FOLLOWER
I20260925 10:25:36.766808 16413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } }
I20260925 10:25:36.767055 16413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:36.767203 16413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:36.767369 16413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:36.769868 16413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } }
I20260925 10:25:36.770392 16413 leader_election.cc:302] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f20e4cd1f446b48d8c75015fd2c4ad; no voters:
I20260925 10:25:36.771040 16413 leader_election.cc:288] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:36.771170 16418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:36.772012 16418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [term 1 LEADER]: Becoming Leader. State: Replica: 23f20e4cd1f446b48d8c75015fd2c4ad, State: Running, Role: LEADER
I20260925 10:25:36.772615 16418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } }
I20260925 10:25:36.773195 16413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:36.776029 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f20e4cd1f446b48d8c75015fd2c4ad. Latest consensus state: current_term: 1 leader_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } } }
I20260925 10:25:36.776432 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:36.776925 16419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f20e4cd1f446b48d8c75015fd2c4ad" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44885 } } }
I20260925 10:25:36.777204 16419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:36.777366 16427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:36.785415 16427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:36.795063 16427 catalog_manager.cc:1347] Generated new cluster ID: ff638f55e6cb4bf99643b30295bcc4dc
I20260925 10:25:36.795190 16427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:36.815480 16427 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:36.816728 16427 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:36.821956 16427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f20e4cd1f446b48d8c75015fd2c4ad: Generated new TSK 0
I20260925 10:25:36.822748 16427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:36.886089 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:37.243068 16439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:37.243393 16439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:37.243490 16439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:37.250994 16439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:37.251104 16439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:37.251219 16439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:37.260948 16439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:37.262374 16439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:37.263708 16439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:37.277593 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:37.277540 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:37.277498 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:37.278451 16439 server_base.cc:1034] running on GCE node
I20260925 10:25:37.279194 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:37.280206 16439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:37.281483 16439 hybrid_clock.cc:648] HybridClock initialized: now 1790331937281413 us; error 55 us; skew 500 ppm
I20260925 10:25:37.281837 16439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:37 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331937, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:37.285046 16439 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:37.286691 16439 webserver.cc:460] Webserver started at http://127.12.248.65:36377/ using document root <none> and password file <none>
I20260925 10:25:37.287389 16439 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:37.287541 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:37.287859 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:37.290931 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "97a6826c5ce246dc8552d27aa8076a4b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "1349844fba25a81254472499c2af37d7"
server_key_iv: "62799b70071fb5807eb84cb899d569bf"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.291745 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "97a6826c5ce246dc8552d27aa8076a4b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "1349844fba25a81254472499c2af37d7"
server_key_iv: "62799b70071fb5807eb84cb899d569bf"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.297008 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:25:37.300642 16454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.302280 16439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:37.302470 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "97a6826c5ce246dc8552d27aa8076a4b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "1349844fba25a81254472499c2af37d7"
server_key_iv: "62799b70071fb5807eb84cb899d569bf"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.302681 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:37.317662 16439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:37.322157 16439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:37.322455 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:37.323387 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:37.324855 16439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:37.324958 16439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.325106 16439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:37.325184 16439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.358767 16439 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36157
I20260925 10:25:37.358803 16566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36157 every 8 connection(s)
I20260925 10:25:37.360112 16439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:37.368319 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16439
I20260925 10:25:37.368525 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:37.369279 13281 external_mini_cluster.cc:1428] Setting key 3963ae65900f82387e6d0eb3e8851dfd
Sep 25 10:25:37 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331937, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:37.381723 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:37.362809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:35419 (local address 127.12.248.126:44885)
0925 10:25:37.363269 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:37.363274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:37.364257 (+ 983us) server_negotiation.cc:407] Connection header received
0925 10:25:37.366160 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:37.366165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:37.366260 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:37.366376 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:37.367820 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.368950 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.370780 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.371252 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.371385 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:37.374882 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:37.374891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:37.374899 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:37.374948 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:37.378049 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:37.379075 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:37.379080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:37.379083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:37.379216 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:37.379720 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:37.379723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:37.379725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:37.379993 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:37.380113 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:37.381082 (+ 969us) server_negotiation.cc:299] Negotiation successful
0925 10:25:37.381273 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260925 10:25:37.383114 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:37.363328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44885 (local address 127.12.248.65:35419)
0925 10:25:37.364042 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:37.364085 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:37.365870 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:37.366608 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:37.366617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:37.367110 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:37.367656 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:37.367671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.369114 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.369118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:37.370632 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:37.370646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.371461 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.371466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:37.371662 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:37.372349 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:37.372372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:37.374628 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:37.378199 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:37.378211 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:37.378257 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:37.378911 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:37.379330 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:37.379338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:37.379344 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:37.379600 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:37.380150 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:37.380162 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:37.380671 (+ 509us) client_negotiation.cc:769] Sending connection context
0925 10:25:37.380976 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 10:25:37.381610 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":279,"threads_started":1}
I20260925 10:25:37.384819 16567 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44885
I20260925 10:25:37.385228 16567 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:37.386111 16567 heartbeater.cc:511] Master 127.12.248.126:44885 requested a full tablet report, sending...
I20260925 10:25:37.388330 16378 ts_manager.cc:194] Registered new tserver with Master: 97a6826c5ce246dc8552d27aa8076a4b (127.12.248.65:36157)
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:37.389981 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:35419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:37.417270 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:37.739550 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:37.739876 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:37.740043 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:37.747344 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:37.747491 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:37.747674 16573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:37.757472 16573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:37.758823 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:37.760196 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:37.773945 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:37.773919 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:37.774053 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:37.774852 16573 server_base.cc:1034] running on GCE node
I20260925 10:25:37.775524 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:37.776580 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:37.777839 16573 hybrid_clock.cc:648] HybridClock initialized: now 1790331937777790 us; error 44 us; skew 500 ppm
I20260925 10:25:37.778174 16573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:37 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331937, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:37.781388 16573 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:37.782922 16573 webserver.cc:460] Webserver started at http://127.12.248.66:34835/ using document root <none> and password file <none>
I20260925 10:25:37.783607 16573 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:37.783758 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:37.784106 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:37.787019 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "9b86f9c6c67549e081b65b6b57da529b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "2d01c8238898c48862db557a9c733707"
server_key_iv: "a51270f0d5fda36cc3de5b0e11ecb982"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.787740 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "9b86f9c6c67549e081b65b6b57da529b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "2d01c8238898c48862db557a9c733707"
server_key_iv: "a51270f0d5fda36cc3de5b0e11ecb982"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.792629 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:37.795899 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.797217 16573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:37.797394 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "9b86f9c6c67549e081b65b6b57da529b"
format_stamp: "Formatted at 2026-09-25 10:25:37 on dist-test-slave-f9rl"
server_key: "2d01c8238898c48862db557a9c733707"
server_key_iv: "a51270f0d5fda36cc3de5b0e11ecb982"
server_key_version: "encryptionkey@0"
I20260925 10:25:37.797595 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:37.816039 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:37.820214 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:37.820535 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:37.821601 16573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:37.823220 16573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:37.823292 16573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.823416 16573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:37.823474 16573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:37.857790 16573 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:41575
I20260925 10:25:37.857838 16700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:41575 every 8 connection(s)
I20260925 10:25:37.859086 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:37.865933 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16573
I20260925 10:25:37.866156 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:37.866634 13281 external_mini_cluster.cc:1428] Setting key 072be209a2b2eea248f17f50b6591d2d
Sep 25 10:25:37 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331937, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:37.884081 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:37.861883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53093 (local address 127.12.248.126:44885)
0925 10:25:37.862098 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:37.862105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:37.863355 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 10:25:37.865226 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:37.865231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:37.865309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:37.865458 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:37.867286 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.868316 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.871100 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.871527 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.871636 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:37.876391 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:37.876401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:37.876409 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:37.876458 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:37.880330 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:37.881259 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:37.881267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:37.881273 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:37.881491 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:37.881983 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:37.881991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:37.881996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:37.882423 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:37.882592 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:37.883341 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 10:25:37.883617 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:37.885126 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:37.862427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44885 (local address 127.12.248.66:53093)
0925 10:25:37.863187 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:37.863232 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:37.864879 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:37.865732 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:37.865741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:37.866437 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:37.867072 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:37.867091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.868542 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.868547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:37.870904 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:37.870926 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:37.871813 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:37.871823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:37.872206 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:37.873018 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:37.873062 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:37.876111 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:37.880509 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:37.880517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:37.880533 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:37.881097 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:37.881652 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:37.881655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:37.881658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:37.881858 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:37.882575 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:37.882582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:37.882931 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 10:25:37.883172 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 10:25:37.883609 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":284,"threads_started":1}
I20260925 10:25:37.887066 16701 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44885
I20260925 10:25:37.887454 16701 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:37.888230 16701 heartbeater.cc:511] Master 127.12.248.126:44885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:37.889854 16378 ts_manager.cc:194] Registered new tserver with Master: 9b86f9c6c67549e081b65b6b57da529b (127.12.248.66:41575)
I20260925 10:25:37.890672 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:53093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:37.914734 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:38.238067 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:38.238340 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:38.238435 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:38.245620 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:38.245734 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:38.245864 16706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:38.256314 16706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:38.257715 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:38.259020 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:38.272617 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:38.272655 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:38.272709 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:38.273545 16706 server_base.cc:1034] running on GCE node
I20260925 10:25:38.274197 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:38.275230 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:38.276466 16706 hybrid_clock.cc:648] HybridClock initialized: now 1790331938276425 us; error 55 us; skew 500 ppm
I20260925 10:25:38.276794 16706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:38.280100 16706 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:38.281749 16706 webserver.cc:460] Webserver started at http://127.12.248.67:39069/ using document root <none> and password file <none>
I20260925 10:25:38.282469 16706 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:38.282616 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:38.282930 16706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:38.285928 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "fea1949626e0434aba5435985dee5a26"
format_stamp: "Formatted at 2026-09-25 10:25:38 on dist-test-slave-f9rl"
server_key: "d72db31de48e68703232f553c888fe2d"
server_key_iv: "f96db799a55d0f54f39c9aae936fdd1a"
server_key_version: "encryptionkey@0"
I20260925 10:25:38.286717 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "fea1949626e0434aba5435985dee5a26"
format_stamp: "Formatted at 2026-09-25 10:25:38 on dist-test-slave-f9rl"
server_key: "d72db31de48e68703232f553c888fe2d"
server_key_iv: "f96db799a55d0f54f39c9aae936fdd1a"
server_key_version: "encryptionkey@0"
I20260925 10:25:38.291814 16706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:38.295432 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:38.296746 16706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:38.296923 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "fea1949626e0434aba5435985dee5a26"
format_stamp: "Formatted at 2026-09-25 10:25:38 on dist-test-slave-f9rl"
server_key: "d72db31de48e68703232f553c888fe2d"
server_key_iv: "f96db799a55d0f54f39c9aae936fdd1a"
server_key_version: "encryptionkey@0"
I20260925 10:25:38.297122 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:38.315742 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:38.320188 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:38.320529 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:38.321579 16706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:38.322921 16706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:38.323025 16706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:38.323173 16706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:38.323253 16706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:38.356889 16706 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:33899
I20260925 10:25:38.356926 16833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:33899 every 8 connection(s)
I20260925 10:25:38.358309 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:38.362937 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16706
I20260925 10:25:38.363121 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:38.363739 13281 external_mini_cluster.cc:1428] Setting key fd079937cea4425a1818df79e2a2d407
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:38.379433 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.360996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:57927 (local address 127.12.248.126:44885)
0925 10:25:38.361226 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:38.361230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:38.362639 (+ 1409us) server_negotiation.cc:407] Connection header received
0925 10:25:38.365159 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:38.365164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:38.365257 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:38.365367 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:38.367056 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.367814 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.369329 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.369778 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.369908 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.373209 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:38.373214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:38.373217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:38.373256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:38.376097 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.376767 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.376771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.376773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.376916 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.377523 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.377528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.377530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.377849 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:38.377993 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:38.378954 (+ 961us) server_negotiation.cc:299] Negotiation successful
0925 10:25:38.379180 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 10:25:38.380403 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.361586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44885 (local address 127.12.248.67:57927)
0925 10:25:38.362423 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:38.362510 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:38.364845 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:38.365669 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:38.365684 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:38.366258 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:38.366896 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:38.366913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.367966 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.367971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:38.369172 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:38.369184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.370013 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.370020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:38.370244 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.370864 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:38.370889 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:38.373010 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:38.376239 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:38.376247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:38.376264 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:38.376668 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:38.377036 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:38.377041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:38.377045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:38.377358 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:38.378065 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:38.378071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:38.378501 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 10:25:38.378738 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 10:25:38.379229 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260925 10:25:38.381942 16834 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44885
I20260925 10:25:38.382257 16834 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:38.382958 16834 heartbeater.cc:511] Master 127.12.248.126:44885 requested a full tablet report, sending...
I20260925 10:25:38.384290 16378 ts_manager.cc:194] Registered new tserver with Master: fea1949626e0434aba5435985dee5a26 (127.12.248.67:33899)
I20260925 10:25:38.385089 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:57927
I20260925 10:25:38.391388 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:25:38.393468 16567 heartbeater.cc:503] Master 127.12.248.126:44885 was elected leader, sending a full tablet report...
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:38.440368 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.426489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.12.248.126:44885)
0925 10:25:38.426666 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:38.426670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:38.427128 (+ 458us) server_negotiation.cc:407] Connection header received
0925 10:25:38.427280 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:38.427283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:38.427347 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:38.427438 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:38.428788 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.429588 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.430811 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.431196 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.431335 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.435213 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:38.435218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:38.435222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:38.435255 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:38.437685 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.438344 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.438348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.438350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.438472 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.438798 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.438801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.438802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.439114 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:38.439294 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:38.439787 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 10:25:38.440010 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 10:25:38.441550 16378 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58456
I20260925 10:25:38.444726 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16439
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:38.462365 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.446235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.12.248.65:36157)
0925 10:25:38.448481 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:38.448488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:38.448524 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:25:38.448581 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:38.448587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:38.448816 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:38.449035 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:38.450355 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.451352 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.452579 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.453139 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.453267 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.457078 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:38.457089 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:38.457103 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:38.457140 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:38.459787 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.460388 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.460395 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.460397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.460510 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.460810 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.460814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.460815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.461131 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:38.461356 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:38.461862 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 10:25:38.462060 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1936,"thread_start_us":1790,"threads_started":1}
W20260925 10:25:38.463662 16562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49084
W20260925 10:25:38.464778 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:38.464967 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16439
I20260925 10:25:38.481184 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16573
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:38.497826 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.482186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.12.248.66:41575)
0925 10:25:38.482800 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:38.482807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:38.482835 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:25:38.482903 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:38.482910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:38.483121 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:38.483264 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:38.484570 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.485677 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.486944 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.487494 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.487594 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.492122 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:38.492135 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:38.492150 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:38.492190 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:38.495256 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.495974 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.495981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.495983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.496112 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.496450 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.496453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.496455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.496790 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:38.497011 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:38.497261 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:25:38.497522 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":299,"threads_started":1}
W20260925 10:25:38.499096 16696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38300
W20260925 10:25:38.500213 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16573: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:38.500378 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16573
I20260925 10:25:38.515763 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16706
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:38.531976 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:38.516759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.12.248.67:33899)
0925 10:25:38.517515 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:38.517532 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:38.517565 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:25:38.517630 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:38.517637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:38.517850 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:38.517982 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:38.519259 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.520307 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.521599 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:38.522170 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:38.522274 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:38.526451 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:38.526462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:38.526478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:38.526515 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:38.529360 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.530179 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.530186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.530189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.530324 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:38.530652 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:38.530655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:38.530658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:38.530997 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:38.531238 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:38.531483 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:25:38.531681 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":363,"threads_started":1}
W20260925 10:25:38.533363 16829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52252
W20260925 10:25:38.534528 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:25:38.534675 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16706
I20260925 10:25:38.549877 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16346
I20260925 10:25:38.551103 16408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:38.656667 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16346
2026-09-25T10:25:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3695 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16851](info): set up 2 sockets
Sep 25 10:25:38 dist-test-slave-f9rl krb5kdc[16851](info): commencing operation
krb5kdc: starting...
W20260925 10:25:39.779269 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:39 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:39.981676 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38265
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:40.312028 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:40.312362 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:40.312456 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:40.319922 16864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:40.320035 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:40.320132 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:40.320217 16864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:40.320302 16864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:40.330189 16864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:40.331642 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:40.332994 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:40.345374 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:40.345510 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:40.345489 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:40.346240 16864 server_base.cc:1034] running on GCE node
I20260925 10:25:40.347064 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:40.348507 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:40.349738 16864 hybrid_clock.cc:648] HybridClock initialized: now 1790331940349676 us; error 53 us; skew 500 ppm
I20260925 10:25:40.350092 16864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:40 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331940, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:40.359442 16864 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:40.361063 16864 webserver.cc:460] Webserver started at http://127.12.248.126:41697/ using document root <none> and password file <none>
I20260925 10:25:40.361876 16864 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:40.362027 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:40.362347 16864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:40.365355 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "473747a0fd0a467b81a7710241a48beb"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "e5ff52928c1b8ff11c64b9dce821d972"
server_key_iv: "1cfcc46cc4904d63ccd29c9de5854e3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.366242 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "473747a0fd0a467b81a7710241a48beb"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "e5ff52928c1b8ff11c64b9dce821d972"
server_key_iv: "1cfcc46cc4904d63ccd29c9de5854e3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.371373 16864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:40.374897 16880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:40.376282 16864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:40.376475 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "473747a0fd0a467b81a7710241a48beb"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "e5ff52928c1b8ff11c64b9dce821d972"
server_key_iv: "1cfcc46cc4904d63ccd29c9de5854e3f"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.376683 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:40.396157 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:40.400408 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:40.400698 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:40.419379 16864 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:38265
I20260925 10:25:40.419402 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:38265 every 8 connection(s)
I20260925 10:25:40.420856 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:40.424724 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:40.430796 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16864
I20260925 10:25:40.431027 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:40.431622 13281 external_mini_cluster.cc:1428] Setting key cfd578b8a631a5db364e93f6c20bf358
I20260925 10:25:40.436405 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb: Bootstrap starting.
I20260925 10:25:40.440236 16933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:40.441502 16933 log.cc:824] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:40.444787 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb: No bootstrap required, opened a new log
Sep 25 10:25:40 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:40.455739 16933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } }
I20260925 10:25:40.456221 16933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:40.456338 16933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473747a0fd0a467b81a7710241a48beb, State: Initialized, Role: FOLLOWER
I20260925 10:25:40.457027 16933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } }
I20260925 10:25:40.457312 16933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:40.457481 16933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:40.457674 16933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:40.457836 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:40.433758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.12.248.126:38265)
0925 10:25:40.434527 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:40.434542 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:40.434594 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:25:40.436968 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:40.436997 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:40.437509 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:40.437947 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:40.439321 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:40.441337 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:40.443111 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:40.444145 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:40.444347 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:40.448878 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:40.448890 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:40.448915 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:40.448972 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:40.452970 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:40.453845 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:40.453855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:40.453864 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:40.454095 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:40.454466 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:40.454472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:40.454477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:40.455203 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:40.455504 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:40.455848 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:25:40.456345 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260925 10:25:40.460126 16933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } }
I20260925 10:25:40.460695 16933 leader_election.cc:302] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473747a0fd0a467b81a7710241a48beb; no voters:
I20260925 10:25:40.461349 16933 leader_election.cc:288] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:40.461509 16938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:40.462349 16938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [term 1 LEADER]: Becoming Leader. State: Replica: 473747a0fd0a467b81a7710241a48beb, State: Running, Role: LEADER
I20260925 10:25:40.462867 16938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } }
I20260925 10:25:40.463459 16933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:40.466226 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473747a0fd0a467b81a7710241a48beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } } }
I20260925 10:25:40.466224 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473747a0fd0a467b81a7710241a48beb. Latest consensus state: current_term: 1 leader_uuid: "473747a0fd0a467b81a7710241a48beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473747a0fd0a467b81a7710241a48beb" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38265 } } }
I20260925 10:25:40.466660 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:40.466660 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:40.468969 16946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:40.473662 16946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:40.483000 16946 catalog_manager.cc:1347] Generated new cluster ID: 6b61d7848fe9456c8df11e90d18bdcf1
I20260925 10:25:40.483137 16946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:40.503827 16946 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:40.505268 16946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:40.518395 16946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473747a0fd0a467b81a7710241a48beb: Generated new TSK 0
I20260925 10:25:40.519290 16946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:40.569765 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:40.909008 16959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:40.909335 16959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:40.909536 16959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:40.917579 16959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:40.917729 16959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:40.917905 16959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:40.927814 16959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:40.929203 16959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:40.930574 16959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:40.944340 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:40.944419 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:40.944391 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:40.945672 16959 server_base.cc:1034] running on GCE node
I20260925 10:25:40.946507 16959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:40.947677 16959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:40.948916 16959 hybrid_clock.cc:648] HybridClock initialized: now 1790331940948861 us; error 47 us; skew 500 ppm
I20260925 10:25:40.949292 16959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:40 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331940, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:40.952646 16959 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:40.954090 16959 webserver.cc:460] Webserver started at http://127.12.248.65:36375/ using document root <none> and password file <none>
I20260925 10:25:40.954774 16959 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:40.954870 16959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:40.955125 16959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:40.958199 16959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "a4973eab526a4c3ca61d86a834682d7b"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "47ed0126f9bc3b8e95ebcb0dd5a706f6"
server_key_iv: "38fbea8f6c26d7acce5a91974fceb9ac"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.958967 16959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "a4973eab526a4c3ca61d86a834682d7b"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "47ed0126f9bc3b8e95ebcb0dd5a706f6"
server_key_iv: "38fbea8f6c26d7acce5a91974fceb9ac"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.964218 16959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:40.968158 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:40.969655 16959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:25:40.969832 16959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "a4973eab526a4c3ca61d86a834682d7b"
format_stamp: "Formatted at 2026-09-25 10:25:40 on dist-test-slave-f9rl"
server_key: "47ed0126f9bc3b8e95ebcb0dd5a706f6"
server_key_iv: "38fbea8f6c26d7acce5a91974fceb9ac"
server_key_version: "encryptionkey@0"
I20260925 10:25:40.970037 16959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:40.989355 16959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:40.993958 16959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:40.994272 16959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:40.995342 16959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:40.996855 16959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:40.996932 16959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:40.997063 16959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:40.997125 16959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.031097 16959 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:35791
I20260925 10:25:41.031137 17087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:35791 every 8 connection(s)
I20260925 10:25:41.032450 16959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:41.040140 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 16959
I20260925 10:25:41.040365 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:41.040992 13281 external_mini_cluster.cc:1428] Setting key 6dc72b0cd39611a4bfc1e127ff8d2cdc
Sep 25 10:25:41 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331940, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:41.053228 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:41.035065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:33473 (local address 127.12.248.126:38265)
0925 10:25:41.035550 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:41.035554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:41.036402 (+ 848us) server_negotiation.cc:407] Connection header received
0925 10:25:41.038514 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:41.038518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:41.038588 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:41.038677 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:41.040180 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.041016 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.043186 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.043594 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.043738 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:41.047318 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:41.047325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:41.047336 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:41.047369 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:41.050034 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:41.050827 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:41.050831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:41.050834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:41.050953 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:41.051414 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:41.051417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:41.051420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:41.051692 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:41.051858 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:41.052608 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 10:25:41.052851 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":266,"threads_started":1}
I20260925 10:25:41.054281 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:41.035513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38265 (local address 127.12.248.65:33473)
0925 10:25:41.036218 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:41.036275 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:41.038164 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:41.038923 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:41.038932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:41.039444 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:41.039996 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:41.040012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.041214 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.041218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:41.043044 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:41.043058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.043833 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.043842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:41.044102 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:41.044809 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:41.044836 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:41.047082 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:41.050191 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:41.050201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:41.050220 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:41.050672 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:41.051068 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:41.051073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:41.051077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:41.051281 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:41.051842 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:41.051851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:41.052215 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 10:25:41.052507 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 10:25:41.053053 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":276,"threads_started":1}
I20260925 10:25:41.055959 17088 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38265
I20260925 10:25:41.056336 17088 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:41.057327 17088 heartbeater.cc:511] Master 127.12.248.126:38265 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:41.059873 16898 ts_manager.cc:194] Registered new tserver with Master: a4973eab526a4c3ca61d86a834682d7b (127.12.248.65:35791)
I20260925 10:25:41.061509 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:33473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:41.085481 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:41.412060 17094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:41.412357 17094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:41.412473 17094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:41.419778 17094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:41.419938 17094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:41.420172 17094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:41.430290 17094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:41.431586 17094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:41.432953 17094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:41.446906 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:41.446951 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:41.447009 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:41.448055 17094 server_base.cc:1034] running on GCE node
I20260925 10:25:41.448746 17094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:41.449869 17094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:41.451207 17094 hybrid_clock.cc:648] HybridClock initialized: now 1790331941451151 us; error 42 us; skew 500 ppm
I20260925 10:25:41.451601 17094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:41 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331941, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:41.455133 17094 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:41.456952 17094 webserver.cc:460] Webserver started at http://127.12.248.66:44163/ using document root <none> and password file <none>
I20260925 10:25:41.457819 17094 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:41.457978 17094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:41.458312 17094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:41.461681 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "01f5e2adbc054ac59b9cfc3cbeb5b05f"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "53b26bdca52ecf7647d79935da577c76"
server_key_iv: "9e004e495e854036cb7e920677367476"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.462595 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "01f5e2adbc054ac59b9cfc3cbeb5b05f"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "53b26bdca52ecf7647d79935da577c76"
server_key_iv: "9e004e495e854036cb7e920677367476"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.468528 17094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:25:41.472308 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.473749 17094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:41.473923 17094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "01f5e2adbc054ac59b9cfc3cbeb5b05f"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "53b26bdca52ecf7647d79935da577c76"
server_key_iv: "9e004e495e854036cb7e920677367476"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.474131 17094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:41.490916 17094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:41.494985 17094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:41.495288 17094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:41.496284 17094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:41.497720 17094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:41.497833 17094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.497998 17094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:41.498071 17094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.532542 17094 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:40773
I20260925 10:25:41.532585 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:40773 every 8 connection(s)
I20260925 10:25:41.534003 17094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:41.538321 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17094
I20260925 10:25:41.538545 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:41.539081 13281 external_mini_cluster.cc:1428] Setting key 799841f68f04e55c6dfdb31ff07d565c
Sep 25 10:25:41 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331941, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:41.558472 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:41.536767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53713 (local address 127.12.248.126:38265)
0925 10:25:41.537011 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:41.537019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:41.538583 (+ 1564us) server_negotiation.cc:407] Connection header received
0925 10:25:41.540980 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:41.540989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:41.541146 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:41.541285 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:41.543574 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.544328 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.546912 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.547428 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.547536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:41.551263 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:41.551272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:41.551280 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:41.551325 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:41.554846 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:41.555745 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:41.555753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:41.555760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:41.555923 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:41.556403 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:41.556408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:41.556412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:41.556827 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:41.557015 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:41.557787 (+ 772us) server_negotiation.cc:299] Negotiation successful
0925 10:25:41.558055 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 10:25:41.559178 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:41.537385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38265 (local address 127.12.248.66:53713)
0925 10:25:41.538374 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:41.538437 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:41.540636 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:41.541692 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:41.541704 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:41.542500 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:41.543370 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:41.543401 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.544507 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.544515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:41.546737 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:41.546753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:41.547671 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:41.547679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:41.548013 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:41.548765 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:41.548812 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:41.551006 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:41.554991 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:41.554998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:41.555011 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:41.555587 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:41.556058 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:41.556063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:41.556069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:41.556267 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:41.556964 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:41.556970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:41.557330 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 10:25:41.557624 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 10:25:41.558020 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:41.560693 17223 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38265
I20260925 10:25:41.560968 17223 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:41.561828 17223 heartbeater.cc:511] Master 127.12.248.126:38265 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:25:41.563436 16898 ts_manager.cc:194] Registered new tserver with Master: 01f5e2adbc054ac59b9cfc3cbeb5b05f (127.12.248.66:40773)
I20260925 10:25:41.564743 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:53713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:41.589615 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:41.909866 17228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:41.910213 17228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:41.910389 17228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:41.917611 17228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:41.917809 17228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:41.918007 17228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:41.927845 17228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:41.929320 17228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:41.930677 17228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:41.943785 17234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:41.944041 17233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:41.944200 17236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:41.944842 17228 server_base.cc:1034] running on GCE node
I20260925 10:25:41.945503 17228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:41.946485 17228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:41.947713 17228 hybrid_clock.cc:648] HybridClock initialized: now 1790331941947673 us; error 71 us; skew 500 ppm
I20260925 10:25:41.948053 17228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:41 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331941, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:41.951859 17228 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:41.953534 17228 webserver.cc:460] Webserver started at http://127.12.248.67:34463/ using document root <none> and password file <none>
I20260925 10:25:41.954217 17228 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:41.954321 17228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:41.954569 17228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:41.957607 17228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "b8f1e058b2724c3a91e0b958e2d5c411"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "3d216e5f0cbb90bcc65cac5451578d6c"
server_key_iv: "cb4f391c40ed344e3833cac139506d6a"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.958353 17228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "b8f1e058b2724c3a91e0b958e2d5c411"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "3d216e5f0cbb90bcc65cac5451578d6c"
server_key_iv: "cb4f391c40ed344e3833cac139506d6a"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.963549 17228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:41.967298 17243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.968796 17228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:41.968935 17228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "b8f1e058b2724c3a91e0b958e2d5c411"
format_stamp: "Formatted at 2026-09-25 10:25:41 on dist-test-slave-f9rl"
server_key: "3d216e5f0cbb90bcc65cac5451578d6c"
server_key_iv: "cb4f391c40ed344e3833cac139506d6a"
server_key_version: "encryptionkey@0"
I20260925 10:25:41.969117 17228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:41.988161 17228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:41.992581 17228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:41.992822 17228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:41.993767 17228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:41.995051 17228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:41.995126 17228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:41.995210 17228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:41.995262 17228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:42.028380 17228 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:35587
I20260925 10:25:42.028422 17356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:35587 every 8 connection(s)
I20260925 10:25:42.029717 17228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:42.038400 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17228
I20260925 10:25:42.038617 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:42.039052 13281 external_mini_cluster.cc:1428] Setting key 170b44752691ba96ec76867e7b7da746
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331941, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:42.050748 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.032694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:50455 (local address 127.12.248.126:38265)
0925 10:25:42.032928 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.032935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.034186 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 10:25:42.036689 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.036694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.036794 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.036882 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.038721 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.039561 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.041561 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.041973 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.042123 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.045507 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.045513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.045516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.045547 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.047981 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.048672 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.048676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.048678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.048807 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.049178 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.049180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.049182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.049514 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.049625 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.050317 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.050485 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 10:25:42.051761 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.033158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38265 (local address 127.12.248.67:50455)
0925 10:25:42.034007 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:42.034048 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:42.036377 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:42.037101 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:42.037109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:42.037698 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:42.038455 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:42.038482 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.039715 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.039725 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:42.041382 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:42.041394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.042190 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.042195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:42.042507 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.043098 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:42.043120 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:42.045248 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:42.048120 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:42.048127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:42.048142 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:42.048564 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:42.048900 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:42.048903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:42.048905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:42.049081 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:42.049619 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:42.049625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:42.049969 (+ 344us) client_negotiation.cc:769] Sending connection context
0925 10:25:42.050211 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 10:25:42.050613 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":332,"threads_started":1}
I20260925 10:25:42.053309 17357 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38265
I20260925 10:25:42.053666 17357 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:42.054402 17357 heartbeater.cc:511] Master 127.12.248.126:38265 requested a full tablet report, sending...
I20260925 10:25:42.055855 16898 ts_manager.cc:194] Registered new tserver with Master: b8f1e058b2724c3a91e0b958e2d5c411 (127.12.248.67:35587)
I20260925 10:25:42.056654 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:50455
I20260925 10:25:42.065069 17088 heartbeater.cc:503] Master 127.12.248.126:38265 was elected leader, sending a full tablet report...
I20260925 10:25:42.067368 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:42.115053 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.100589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.12.248.65:35791)
0925 10:25:42.101237 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.101245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.101272 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:25:42.101333 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.101339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.101628 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.101765 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.103092 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.104097 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.105234 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.105857 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.105951 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.109838 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.109849 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.109863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.109895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.112522 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.113142 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.113150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.113152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.113287 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.113674 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.113677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.113679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.114005 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.114257 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.114459 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.114647 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":343,"threads_started":1}
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:42.134430 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.120345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.12.248.126:38265)
0925 10:25:42.120530 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.120535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.120598 (+ 63us) server_negotiation.cc:407] Connection header received
0925 10:25:42.120773 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.120777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.120835 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.120916 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.122465 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.123181 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.124575 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.124989 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.125089 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.129133 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.129138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.129141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.129175 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.131790 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.132481 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.132485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.132487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.132605 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.132963 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.132968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.132970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.133377 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.133736 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.133951 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.134139 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:25:42.135140 16897 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48490
I20260925 10:25:42.136307 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16959
I20260925 10:25:42.149293 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.137302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.12.248.65:35791)
0925 10:25:42.137538 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.137542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.137936 (+ 394us) server_negotiation.cc:407] Connection header received
0925 10:25:42.138054 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.138060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.138197 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.138331 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.139550 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.140692 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.141979 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.142386 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.142505 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.144685 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.144690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.144693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.144723 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.147155 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.147811 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.147815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.147817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.147931 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.148235 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.148238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.148240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.148472 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.148641 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.148899 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.149057 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:25:42.149940 17083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:42.231784 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16959
I20260925 10:25:42.248088 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17094
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:42.263859 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.248915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.12.248.66:40773)
0925 10:25:42.249550 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.249557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.249585 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:25:42.249653 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.249659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.249890 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.250030 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.251219 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.252201 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.253734 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.254283 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.254383 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.258493 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.258505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.258519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.258549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.261307 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.262045 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.262052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.262055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.262172 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.262505 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.262508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.262510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.262847 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.263061 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.263322 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.263549 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260925 10:25:42.265271 17218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:42.349330 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17094
I20260925 10:25:42.366108 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17228
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:42.382385 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:42.367222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.12.248.67:35587)
0925 10:25:42.367901 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:42.367908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:42.367936 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:25:42.367997 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:42.368003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:42.368274 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:42.368409 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:42.369773 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.370789 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.371977 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:42.372641 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:42.372738 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:42.377019 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:42.377029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:42.377044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:42.377076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:42.379848 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.380503 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.380511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.380513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.380628 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:42.380935 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:42.380938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:42.380940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:42.381344 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:42.381714 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:42.381856 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:25:42.382044 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":367,"threads_started":1}
I20260925 10:25:42.383687 17352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:42.465547 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17228
I20260925 10:25:42.481199 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 16864
I20260925 10:25:42.482475 16928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:42.586599 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 16864
2026-09-25T10:25:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3932 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[17376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[17376](info): set up 2 sockets
Sep 25 10:25:42 dist-test-slave-f9rl krb5kdc[17376](info): commencing operation
krb5kdc: starting...
W20260925 10:25:43.683382 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:43 dist-test-slave-f9rl krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:43.885083 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35975
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:35975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:44.217885 17390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:44.218189 17390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:44.218389 17390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:44.225816 17390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:44.225927 17390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:44.226039 17390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:44.226143 17390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:44.226241 17390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:44.236263 17390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:35975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:44.237599 17390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:44.238945 17390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:44.251246 17395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:44.251233 17398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:44.251200 17396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:44.255009 17390 server_base.cc:1034] running on GCE node
I20260925 10:25:44.255832 17390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:44.257313 17390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:44.258518 17390 hybrid_clock.cc:648] HybridClock initialized: now 1790331944258479 us; error 36 us; skew 500 ppm
I20260925 10:25:44.258895 17390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:44 dist-test-slave-f9rl krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:44.261986 17390 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:44.263195 17390 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:44.263330 17390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:44.263670 17390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:44.266819 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "3f0764df965140e9a1a7481344aea9db"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "d60167481e8380cbf6055242df412eb5"
server_key_iv: "2f213a255f049b56009b0c3a8c88f09b"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.267642 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "3f0764df965140e9a1a7481344aea9db"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "d60167481e8380cbf6055242df412eb5"
server_key_iv: "2f213a255f049b56009b0c3a8c88f09b"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.273041 17390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:44.276542 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:44.277921 17390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:44.278107 17390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "3f0764df965140e9a1a7481344aea9db"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "d60167481e8380cbf6055242df412eb5"
server_key_iv: "2f213a255f049b56009b0c3a8c88f09b"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.278340 17390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:44.304497 17390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:44.308781 17390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:44.309067 17390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:44.327605 17390 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:35975
I20260925 10:25:44.327630 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:35975 every 8 connection(s)
I20260925 10:25:44.328549 17390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:44.332340 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:44.335379 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17390
I20260925 10:25:44.335618 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:44.336247 13281 external_mini_cluster.cc:1428] Setting key fc2b4d6234a9aae1dc2f7868f56b049f
I20260925 10:25:44.345932 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: Bootstrap starting.
I20260925 10:25:44.349764 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:44.350804 17456 log.cc:824] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:25:44 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:44.353754 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: No bootstrap required, opened a new log
I20260925 10:25:44.360149 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:44.338412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.12.248.126:35975)
0925 10:25:44.339259 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:44.339277 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:44.339330 (+ 53us) server_negotiation.cc:407] Connection header received
0925 10:25:44.341401 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:44.341456 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:44.341993 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:44.342467 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:44.343976 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.346018 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.347352 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.348444 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.348637 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:44.352756 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:44.352763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:44.352779 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:44.352819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:44.355868 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:44.356568 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:44.356578 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:44.356584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:44.356758 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:44.357148 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:44.357169 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:44.357171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:44.357805 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:44.358137 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:44.358608 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:25:44.359056 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":372,"threads_started":1}
I20260925 10:25:44.361888 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } }
I20260925 10:25:44.362247 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:44.362372 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f0764df965140e9a1a7481344aea9db, State: Initialized, Role: FOLLOWER
I20260925 10:25:44.363066 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } }
I20260925 10:25:44.363340 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:44.363472 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:44.363646 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:44.365939 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } }
I20260925 10:25:44.366436 17456 leader_election.cc:302] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f0764df965140e9a1a7481344aea9db; no voters:
I20260925 10:25:44.367133 17456 leader_election.cc:288] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:44.367324 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:44.368563 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [term 1 LEADER]: Becoming Leader. State: Replica: 3f0764df965140e9a1a7481344aea9db, State: Running, Role: LEADER
I20260925 10:25:44.369066 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } }
I20260925 10:25:44.369472 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:44.371552 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f0764df965140e9a1a7481344aea9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } } }
I20260925 10:25:44.371858 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:44.372723 17466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:44.372592 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f0764df965140e9a1a7481344aea9db. Latest consensus state: current_term: 1 leader_uuid: "3f0764df965140e9a1a7481344aea9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f0764df965140e9a1a7481344aea9db" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35975 } } }
I20260925 10:25:44.373872 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:44.379689 17466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:44.389828 17466 catalog_manager.cc:1347] Generated new cluster ID: d294253e210a415d95b4926bab2463d0
I20260925 10:25:44.389967 17466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:44.410153 17466 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:44.411492 17466 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:44.420868 17466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: Generated new TSK 0
I20260925 10:25:44.421837 17466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:44.477977 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:44.804761 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:44.805055 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:44.805218 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:44.812716 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:44.812836 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:44.813030 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:44.822873 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:44.824234 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:44.825577 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:44.838529 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:44.838496 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:44.838959 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:44.839385 17482 server_base.cc:1034] running on GCE node
I20260925 10:25:44.840027 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:44.841046 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:44.842285 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790331944842243 us; error 56 us; skew 500 ppm
I20260925 10:25:44.842641 17482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:44 dist-test-slave-f9rl krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:44.845844 17482 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:44.847298 17482 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:44.847431 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:44.847788 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:44.851015 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "7c9a5a0801b84868bb7e75fcfb5f024d"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "f9b457c3dc8a27c332a0ee58db00c596"
server_key_iv: "c9da7c21a6977a393ff767e2911e2ac7"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.851850 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "7c9a5a0801b84868bb7e75fcfb5f024d"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "f9b457c3dc8a27c332a0ee58db00c596"
server_key_iv: "c9da7c21a6977a393ff767e2911e2ac7"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.857038 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260925 10:25:44.860853 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:44.862461 17482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:25:44.862769 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "7c9a5a0801b84868bb7e75fcfb5f024d"
format_stamp: "Formatted at 2026-09-25 10:25:44 on dist-test-slave-f9rl"
server_key: "f9b457c3dc8a27c332a0ee58db00c596"
server_key_iv: "c9da7c21a6977a393ff767e2911e2ac7"
server_key_version: "encryptionkey@0"
I20260925 10:25:44.863029 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:44.883857 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:44.888283 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:44.888585 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:44.889683 17482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:44.891145 17482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:44.891276 17482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:44.891443 17482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:44.891541 17482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:44.925894 17482 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:37147
I20260925 10:25:44.925953 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:37147 every 8 connection(s)
I20260925 10:25:44.926789 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:44.928153 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17482
I20260925 10:25:44.928380 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:44.929004 13281 external_mini_cluster.cc:1428] Setting key d39e7de9f6a00de9188ac472f12aefbc
Sep 25 10:25:44 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331944, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:44.947196 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:44.929353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49479 (local address 127.12.248.126:35975)
0925 10:25:44.931504 (+ 2151us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:44.931510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:44.931531 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:25:44.932875 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:44.932881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:44.932971 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:44.933088 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:44.935058 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.935775 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.937352 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.937822 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.937957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:44.941387 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:44.941392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:44.941396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:44.941430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:44.944007 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:44.944748 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:44.944753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:44.944758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:44.944885 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:44.945318 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:44.945324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:44.945327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:44.945668 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:44.945843 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:44.946555 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 10:25:44.946820 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2033,"thread_start_us":1896,"threads_started":1}
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:44.948199 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:44.929919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35975 (local address 127.12.248.65:49479)
0925 10:25:44.930722 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:44.930781 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:44.932595 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:44.933362 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:44.933372 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:44.934237 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:44.934883 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:44.934902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.935937 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.935944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:44.937197 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:44.937209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:44.938068 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:44.938075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:44.938303 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:44.938994 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:44.939018 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:44.941146 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:44.944157 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:44.944166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:44.944185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:44.944613 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:44.945005 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:44.945010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:44.945014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:44.945202 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:44.945821 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:44.945829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:44.946180 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 10:25:44.946461 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 10:25:44.947036 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":345,"threads_started":1}
I20260925 10:25:44.949859 17610 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
I20260925 10:25:44.950234 17610 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:44.950929 17610 heartbeater.cc:511] Master 127.12.248.126:35975 requested a full tablet report, sending...
I20260925 10:25:44.953256 17421 ts_manager.cc:194] Registered new tserver with Master: 7c9a5a0801b84868bb7e75fcfb5f024d (127.12.248.65:37147)
I20260925 10:25:44.955418 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:49479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:44.976421 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:45.307132 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:45.307474 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:45.307657 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:45.315135 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:45.315294 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:45.315488 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:45.325155 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:45.326556 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:45.327948 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:45.341102 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:45.341158 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:45.344308 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:45.344689 17616 server_base.cc:1034] running on GCE node
I20260925 10:25:45.345306 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:45.346334 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:45.347704 17616 hybrid_clock.cc:648] HybridClock initialized: now 1790331945347612 us; error 42 us; skew 500 ppm
I20260925 10:25:45.348161 17616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:45 dist-test-slave-f9rl krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331945, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:45.351667 17616 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:45.352988 17616 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:45.353207 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:45.353641 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:45.356756 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "9f1d0aaa4019451a852ce71b1d9cc243"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "e247dc04f82c08e1bf193e0198522fb6"
server_key_iv: "c6fc3882558ada17697339fc8e0f8758"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.357755 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "9f1d0aaa4019451a852ce71b1d9cc243"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "e247dc04f82c08e1bf193e0198522fb6"
server_key_iv: "c6fc3882558ada17697339fc8e0f8758"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.363408 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:45.367231 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.368727 17616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 10:25:45.368871 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "9f1d0aaa4019451a852ce71b1d9cc243"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "e247dc04f82c08e1bf193e0198522fb6"
server_key_iv: "c6fc3882558ada17697339fc8e0f8758"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.369059 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:45.394958 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:45.399855 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:45.400195 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:45.401244 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:45.402729 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:45.402863 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.403009 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:45.403138 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.439127 17616 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:34797
I20260925 10:25:45.439162 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:34797 every 8 connection(s)
I20260925 10:25:45.440121 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:45.446403 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17616
I20260925 10:25:45.446611 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:45.447170 13281 external_mini_cluster.cc:1428] Setting key c86df62ed20622cb9533142bb278059c
Sep 25 10:25:45 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331945, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:45.464676 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:45.442765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:33151 (local address 127.12.248.126:35975)
0925 10:25:45.442957 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:45.442961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:45.444206 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 10:25:45.446218 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:45.446223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:45.446301 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:45.446400 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:45.448499 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.449607 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.452279 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.452712 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.452831 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:45.457711 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:45.457721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:45.457729 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:45.457782 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:45.461358 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:45.462106 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:45.462114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:45.462121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:45.462277 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:45.462676 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:45.462683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:45.462689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:45.463079 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:45.463235 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:45.463934 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 10:25:45.464203 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:45.465533 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:45.443318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35975 (local address 127.12.248.66:33151)
0925 10:25:45.444039 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:45.444077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:45.445855 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:45.446684 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:45.446696 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:45.447485 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:45.448300 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:45.448335 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.449777 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.449782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:45.452044 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:45.452065 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.452977 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.452985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:45.453335 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:45.454157 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:45.454194 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:45.457413 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:45.461535 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:45.461542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:45.461556 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:45.461979 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:45.462387 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:45.462390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:45.462393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:45.462568 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:45.463200 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:45.463205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:45.463560 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 10:25:45.463794 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 10:25:45.464198 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":305,"threads_started":1}
I20260925 10:25:45.467170 17744 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35975
I20260925 10:25:45.467516 17744 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:45.468410 17744 heartbeater.cc:511] Master 127.12.248.126:35975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:25:45.470172 17421 ts_manager.cc:194] Registered new tserver with Master: 9f1d0aaa4019451a852ce71b1d9cc243 (127.12.248.66:34797)
I20260925 10:25:45.471067 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:33151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:45.497632 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:45.820467 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:45.820755 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:45.820924 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:45.828179 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:45.828286 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:45.828449 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:45.838248 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:45.839569 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:45.840951 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:45.853826 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:45.853922 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:45.853968 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:45.854626 17749 server_base.cc:1034] running on GCE node
I20260925 10:25:45.855240 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:45.856220 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:45.857475 17749 hybrid_clock.cc:648] HybridClock initialized: now 1790331945857382 us; error 45 us; skew 500 ppm
I20260925 10:25:45.857856 17749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:45 dist-test-slave-f9rl krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331945, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:45.860949 17749 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:45.862179 17749 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:45.862323 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:45.862648 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:45.865695 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "53e255590c734aa3950261d1443d9e27"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "802c8c5e45fe7b0912cef809043209b6"
server_key_iv: "570807968fa1d5863503ccf6fbce5d2e"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.866722 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "53e255590c734aa3950261d1443d9e27"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "802c8c5e45fe7b0912cef809043209b6"
server_key_iv: "570807968fa1d5863503ccf6fbce5d2e"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.872220 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:45.875866 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.877384 17749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:25:45.877570 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "53e255590c734aa3950261d1443d9e27"
format_stamp: "Formatted at 2026-09-25 10:25:45 on dist-test-slave-f9rl"
server_key: "802c8c5e45fe7b0912cef809043209b6"
server_key_iv: "570807968fa1d5863503ccf6fbce5d2e"
server_key_version: "encryptionkey@0"
I20260925 10:25:45.877837 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:45.912989 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:45.917270 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:45.917598 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:45.918612 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:45.919970 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:45.920097 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.920264 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:45.920354 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:45.954028 17749 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:32965
I20260925 10:25:45.954087 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:32965 every 8 connection(s)
I20260925 10:25:45.954962 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:45.956843 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17749
I20260925 10:25:45.957068 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:45.957641 13281 external_mini_cluster.cc:1428] Setting key aa06a6746fd4512338e4d2232e18239c
I20260925 10:25:45.958856 17610 heartbeater.cc:503] Master 127.12.248.126:35975 was elected leader, sending a full tablet report...
Sep 25 10:25:45 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331945, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:45.977605 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:45.957632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:42459 (local address 127.12.248.126:35975)
0925 10:25:45.957864 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:45.957870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:45.959292 (+ 1422us) server_negotiation.cc:407] Connection header received
0925 10:25:45.961589 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:45.961594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:45.961714 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:45.961837 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:45.963336 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.964085 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.966345 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.967128 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.967291 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:45.970983 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:45.970989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:45.970993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:45.971032 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:45.974263 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:45.975218 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:45.975222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:45.975224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:45.975356 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:45.975863 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:45.975866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:45.975868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:45.976156 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:45.976278 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:45.977059 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 10:25:45.977290 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:25:45.978559 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:45.958202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35975 (local address 127.12.248.67:42459)
0925 10:25:45.959123 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:45.959188 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:45.961284 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:45.962068 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:45.962076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:45.962578 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:45.963154 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:45.963171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.964234 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.964239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:45.966176 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:45.966192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.967389 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.967397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:45.967692 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:45.968367 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:45.968389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:45.970750 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:45.974600 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:45.974608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:45.974622 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:45.975104 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:45.975499 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:45.975505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:45.975513 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:45.975756 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:45.976275 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:45.976282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:45.976656 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 10:25:45.976951 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 10:25:45.977394 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":397,"threads_started":1}
I20260925 10:25:45.979861 17878 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35975
I20260925 10:25:45.980166 17878 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:45.980805 17878 heartbeater.cc:511] Master 127.12.248.126:35975 requested a full tablet report, sending...
I20260925 10:25:45.982373 17420 ts_manager.cc:194] Registered new tserver with Master: 53e255590c734aa3950261d1443d9e27 (127.12.248.67:32965)
I20260925 10:25:45.983343 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:42459
I20260925 10:25:45.985721 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:25:46.003050 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:45.990271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.12.248.126:35975)
0925 10:25:45.990474 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:45.990483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:45.990949 (+ 466us) server_negotiation.cc:407] Connection header received
0925 10:25:45.991145 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:45.991150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:45.991237 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:45.991350 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:45.992719 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.993510 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.994779 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:45.995218 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:45.995365 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:45.997688 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:45.997694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:45.997699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:45.997740 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:46.000304 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.001016 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.001023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.001027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.001160 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.001567 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.001573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.001576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.001925 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:46.002204 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.002516 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.002728 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:25:46.007282 17420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:46.014303 17420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:46.050689 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.026166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:34797 (local address 127.0.0.1:58418)
0925 10:25:46.037367 (+ 11201us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.037408 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.037632 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.039931 (+ 2299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.039943 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.039986 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.040550 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.040566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.043153 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.043162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.046540 (+ 3378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.046563 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.049662 (+ 3099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.049667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.049888 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.049954 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.050096 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.050315 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10945,"thread_start_us":12317,"threads_started":1}
I20260925 10:25:46.050701 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.026410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.12.248.66:34797)
0925 10:25:46.038888 (+ 12478us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.038895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.038920 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:25:46.039047 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.039059 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.039352 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.039590 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.040800 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.042905 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.046782 (+ 3877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.048878 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.049006 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.049390 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.050172 (+ 782us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.050343 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12327,"thread_start_us":12183,"threads_started":1}
I20260925 10:25:46.052626 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.026873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32965 (local address 127.0.0.1:49094)
0925 10:25:46.038161 (+ 11288us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.038184 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.038331 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.043267 (+ 4936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.043272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.043304 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.043902 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.043919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.047520 (+ 3601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.047528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.049836 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.049852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.051791 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.051798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.051977 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.052012 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.052109 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.052243 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11096,"mutex_wait_us":1,"thread_start_us":10062,"threads_started":1}
I20260925 10:25:46.052706 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.027058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.12.248.67:32965)
0925 10:25:46.042377 (+ 15319us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.042386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.042418 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:25:46.042523 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.042531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.042808 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.044182 (+ 1374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.044537 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.046697 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.049984 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.051582 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.051715 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.051942 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.052188 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.052359 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15082,"thread_start_us":15887,"threads_started":1}
I20260925 10:25:46.055433 17678 tablet_service.cc:1467] Processing CreateTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f (DEFAULT_TABLE table=test-table [id=a8e8086d82664a39ace2b202c54e837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:46.056998 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691d1699bfe2441f9b58bf63d144dc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:46.056560 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f (DEFAULT_TABLE table=test-table [id=a8e8086d82664a39ace2b202c54e837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:46.057593 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.039198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:37147 (local address 127.0.0.1:39614)
0925 10:25:46.047250 (+ 8052us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.047265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.047350 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.050930 (+ 3580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.050943 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.050966 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.051346 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.051362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.053336 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.053341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.054808 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.054821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.056808 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.056814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.056954 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.056979 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.057094 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.057233 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7937,"thread_start_us":7781,"threads_started":1}
I20260925 10:25:46.058058 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.039267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.12.248.65:37147)
0925 10:25:46.050023 (+ 10756us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.050030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.050053 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:25:46.050171 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.050180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.050453 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.050653 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.051570 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.053144 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.054970 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.056283 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.056423 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.056657 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.057333 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.057547 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10552,"thread_start_us":10383,"threads_started":1}
I20260925 10:25:46.058390 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691d1699bfe2441f9b58bf63d144dc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:46.061254 17545 tablet_service.cc:1467] Processing CreateTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f (DEFAULT_TABLE table=test-table [id=a8e8086d82664a39ace2b202c54e837b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:46.062391 17545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691d1699bfe2441f9b58bf63d144dc9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:46.072455 17896 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Bootstrap starting.
I20260925 10:25:46.074985 17897 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Bootstrap starting.
I20260925 10:25:46.076467 17898 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Bootstrap starting.
I20260925 10:25:46.077703 17896 tablet_bootstrap.cc:654] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:46.078666 17897 tablet_bootstrap.cc:654] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:46.078936 17896 log.cc:824] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:46.079885 17897 log.cc:824] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:46.080015 17898 tablet_bootstrap.cc:654] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:46.081226 17898 log.cc:824] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:46.082180 17896 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: No bootstrap required, opened a new log
I20260925 10:25:46.082469 17896 ts_tablet_manager.cc:1397] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 10:25:46.083127 17897 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: No bootstrap required, opened a new log
I20260925 10:25:46.083395 17897 ts_tablet_manager.cc:1397] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260925 10:25:46.087838 17898 tablet_bootstrap.cc:492] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: No bootstrap required, opened a new log
I20260925 10:25:46.088160 17898 ts_tablet_manager.cc:1397] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 10:25:46.093884 17896 raft_consensus.cc:348] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.094344 17896 raft_consensus.cc:374] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:46.094448 17896 raft_consensus.cc:729] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1d0aaa4019451a852ce71b1d9cc243, State: Initialized, Role: FOLLOWER
I20260925 10:25:46.095090 17896 consensus_queue.cc:260] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.095165 17897 raft_consensus.cc:348] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.095585 17897 raft_consensus.cc:374] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:46.095721 17897 raft_consensus.cc:729] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e255590c734aa3950261d1443d9e27, State: Initialized, Role: FOLLOWER
I20260925 10:25:46.096455 17744 heartbeater.cc:503] Master 127.12.248.126:35975 was elected leader, sending a full tablet report...
I20260925 10:25:46.096310 17897 consensus_queue.cc:260] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.097090 17896 ts_tablet_manager.cc:1428] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260925 10:25:46.096940 17898 raft_consensus.cc:348] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.097322 17898 raft_consensus.cc:374] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:46.097409 17898 raft_consensus.cc:729] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9a5a0801b84868bb7e75fcfb5f024d, State: Initialized, Role: FOLLOWER
I20260925 10:25:46.097957 17898 consensus_queue.cc:260] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.099447 17898 ts_tablet_manager.cc:1428] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 10:25:46.100245 17878 heartbeater.cc:503] Master 127.12.248.126:35975 was elected leader, sending a full tablet report...
I20260925 10:25:46.100931 17897 ts_tablet_manager.cc:1428] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Time spent starting tablet: real 0.017s user 0.006s sys 0.012s
W20260925 10:25:46.179119 17611 tablet.cc:2369] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:25:46.192157 17745 tablet.cc:2369] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:46.195724 17903 raft_consensus.cc:482] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:46.196039 17903 raft_consensus.cc:504] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.197707 17903 leader_election.cc:288] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f1d0aaa4019451a852ce71b1d9cc243 (127.12.248.66:34797), 7c9a5a0801b84868bb7e75fcfb5f024d (127.12.248.65:37147)
I20260925 10:25:46.204353 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.198064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:34797 (local address 127.12.248.67:38575)
0925 10:25:46.198267 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.198286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.198405 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.198988 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.198993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.199028 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.199186 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.199194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.200558 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.200563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.202166 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.202179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.203568 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.203574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.203766 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.203801 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.203900 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.204041 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:25:46.204555 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.198315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38575 (local address 127.12.248.66:34797)
0925 10:25:46.198502 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.198508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.198531 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:25:46.198613 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.198619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.198702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.198807 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.199381 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.200420 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.202332 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.203424 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.203546 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.203596 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.203990 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.204134 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:25:46.205134 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.198798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37261 (local address 127.12.248.65:37147)
0925 10:25:46.198992 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.198997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.199251 (+ 254us) server_negotiation.cc:407] Connection header received
0925 10:25:46.199404 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.199408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.199464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.199531 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.200147 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.201128 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.203123 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.204157 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.204305 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.204383 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.204762 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.204897 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:25:46.205158 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.198500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:37147 (local address 127.12.248.67:37261)
0925 10:25:46.199144 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.199162 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.199257 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.199572 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.199576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.199603 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.199979 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.199991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.201272 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.201275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.202966 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.202980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.204362 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.204368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.204562 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.204602 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.204710 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.204842 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":356,"threads_started":1}
I20260925 10:25:46.205296 17902 raft_consensus.cc:482] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:46.205607 17902 raft_consensus.cc:504] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.205651 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "53e255590c734aa3950261d1443d9e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" is_pre_election: true
I20260925 10:25:46.206347 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "53e255590c734aa3950261d1443d9e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" is_pre_election: true
W20260925 10:25:46.206540 17879 tablet.cc:2369] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:46.206532 17699 raft_consensus.cc:2454] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e255590c734aa3950261d1443d9e27 in term 0.
I20260925 10:25:46.206879 17565 raft_consensus.cc:2454] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e255590c734aa3950261d1443d9e27 in term 0.
I20260925 10:25:46.207355 17767 leader_election.cc:302] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e255590c734aa3950261d1443d9e27, 9f1d0aaa4019451a852ce71b1d9cc243; no voters:
I20260925 10:25:46.207773 17903 raft_consensus.cc:2781] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:46.207943 17903 raft_consensus.cc:482] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:46.208218 17903 raft_consensus.cc:3037] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:46.210160 17902 leader_election.cc:288] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c9a5a0801b84868bb7e75fcfb5f024d (127.12.248.65:37147), 53e255590c734aa3950261d1443d9e27 (127.12.248.67:32965)
I20260925 10:25:46.211215 17903 raft_consensus.cc:504] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.211959 17903 leader_election.cc:288] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [CANDIDATE]: Term 1 election: Requested vote from peers 9f1d0aaa4019451a852ce71b1d9cc243 (127.12.248.66:34797), 7c9a5a0801b84868bb7e75fcfb5f024d (127.12.248.65:37147)
I20260925 10:25:46.212482 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "53e255590c734aa3950261d1443d9e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d"
I20260925 10:25:46.212738 17565 raft_consensus.cc:3037] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:46.212661 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "53e255590c734aa3950261d1443d9e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1d0aaa4019451a852ce71b1d9cc243"
I20260925 10:25:46.212950 17699 raft_consensus.cc:3037] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:46.216130 17565 raft_consensus.cc:2454] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e255590c734aa3950261d1443d9e27 in term 1.
I20260925 10:25:46.216717 17767 leader_election.cc:302] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e255590c734aa3950261d1443d9e27, 7c9a5a0801b84868bb7e75fcfb5f024d; no voters:
I20260925 10:25:46.217128 17903 raft_consensus.cc:2781] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:46.217645 17699 raft_consensus.cc:2454] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e255590c734aa3950261d1443d9e27 in term 1.
I20260925 10:25:46.218482 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.208022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:37287 (local address 127.12.248.65:37147)
0925 10:25:46.208167 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.208172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.209037 (+ 865us) server_negotiation.cc:407] Connection header received
0925 10:25:46.209216 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.209232 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.209328 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.209462 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.210067 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.211493 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.214782 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.216225 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.216560 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.216632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.217903 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.218098 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:25:46.219370 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.207849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:37147 (local address 127.12.248.66:37287)
0925 10:25:46.208922 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.208941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.209077 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.209486 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.209492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.209521 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.209910 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.209923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.211667 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.211673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.214561 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.214578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.216944 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.216953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.217166 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.217265 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.217363 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.217586 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":748,"threads_started":1}
I20260925 10:25:46.220290 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" is_pre_election: true
I20260925 10:25:46.220764 17565 raft_consensus.cc:2379] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f1d0aaa4019451a852ce71b1d9cc243 in current term 1: Already voted for candidate 53e255590c734aa3950261d1443d9e27 in this term.
I20260925 10:25:46.221050 17903 raft_consensus.cc:686] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 1 LEADER]: Becoming Leader. State: Replica: 53e255590c734aa3950261d1443d9e27, State: Running, Role: LEADER
I20260925 10:25:46.222599 17903 consensus_queue.cc:237] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } }
I20260925 10:25:46.226039 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.208064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32965 (local address 127.12.248.66:43733)
0925 10:25:46.219281 (+ 11217us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:46.219306 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:46.219674 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:46.220246 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:46.220251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:46.220279 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:25:46.220623 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.220636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.222140 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.222144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.223549 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:46.223561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.225174 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.225180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:46.225339 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.225378 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:25:46.225505 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 10:25:46.225638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11067,"thread_start_us":11334,"threads_started":1}
I20260925 10:25:46.226050 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.208668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:43733 (local address 127.12.248.67:32965)
0925 10:25:46.208825 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.208832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.219393 (+ 10561us) server_negotiation.cc:407] Connection header received
0925 10:25:46.219853 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.219858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.219939 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.220019 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:25:46.220803 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.222015 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.223720 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.224954 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.225129 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.225212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.225568 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.225721 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:25:46.227147 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691d1699bfe2441f9b58bf63d144dc9f" candidate_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e255590c734aa3950261d1443d9e27" is_pre_election: true
I20260925 10:25:46.228188 17633 leader_election.cc:302] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f1d0aaa4019451a852ce71b1d9cc243; no voters: 53e255590c734aa3950261d1443d9e27, 7c9a5a0801b84868bb7e75fcfb5f024d
I20260925 10:25:46.227936 17420 catalog_manager.cc:5495] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e255590c734aa3950261d1443d9e27 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "53e255590c734aa3950261d1443d9e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53e255590c734aa3950261d1443d9e27" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32965 } health_report { overall_health: HEALTHY } } }
I20260925 10:25:46.228642 17902 raft_consensus.cc:2726] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:25:46.284709 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.277282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.12.248.67:32965)
0925 10:25:46.277516 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.277524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.277664 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:25:46.277906 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.277913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.278047 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.278304 (+ 257us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:25:46.278774 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.279989 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.281671 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.282336 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.282460 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.283159 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:25:46.284015 (+ 856us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:25:46.284204 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.284265 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.284366 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:25:46.295984 17565 raft_consensus.cc:1261] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 1 FOLLOWER]: Refusing update from remote peer 53e255590c734aa3950261d1443d9e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:46.296319 17699 raft_consensus.cc:1261] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 1 FOLLOWER]: Refusing update from remote peer 53e255590c734aa3950261d1443d9e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:46.296974 17903 consensus_queue.cc:1035] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9a5a0801b84868bb7e75fcfb5f024d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:46.297366 17908 consensus_queue.cc:1035] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1d0aaa4019451a852ce71b1d9cc243" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:46.310885 17911 mvcc.cc:205] Tried to move back new op lower bound from 7333199652016795648 to 7333199651732410368. Current Snapshot: MvccSnapshot[applied={T|T < 7333199652016795648}]
I20260925 10:25:46.314855 17912 mvcc.cc:205] Tried to move back new op lower bound from 7333199652016795648 to 7333199651732410368. Current Snapshot: MvccSnapshot[applied={T|T < 7333199652016795648}]
I20260925 10:25:46.323891 17419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:46.324189 17419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:46.328284 17419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f0764df965140e9a1a7481344aea9db: Sending DeleteTablet for 3 replicas of tablet 691d1699bfe2441f9b58bf63d144dc9f
I20260925 10:25:46.329779 17545 tablet_service.cc:1514] Processing DeleteTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:46 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:39614
I20260925 10:25:46.329823 17678 tablet_service.cc:1514] Processing DeleteTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:46 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:58418
I20260925 10:25:46.330868 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 691d1699bfe2441f9b58bf63d144dc9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:46 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:49094
I20260925 10:25:46.331429 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17482
I20260925 10:25:46.346041 17921 tablet_replica.cc:330] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: stopping tablet replica
I20260925 10:25:46.346688 17921 raft_consensus.cc:2229] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:46.347354 17921 raft_consensus.cc:2258] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:46.348259 17922 tablet_replica.cc:330] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: stopping tablet replica
I20260925 10:25:46.348667 17922 raft_consensus.cc:2229] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:46.349156 17922 raft_consensus.cc:2258] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:46.351356 17923 tablet_replica.cc:330] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: stopping tablet replica
I20260925 10:25:46.351655 17921 ts_tablet_manager.cc:1905] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:46.351773 17923 raft_consensus.cc:2229] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:46.352526 17923 raft_consensus.cc:2258] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:46.354714 17923 ts_tablet_manager.cc:1905] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:46.359234 17922 ts_tablet_manager.cc:1905] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:46.363848 17923 ts_tablet_manager.cc:1918] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:46.364107 17923 log.cc:1196] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/691d1699bfe2441f9b58bf63d144dc9f
I20260925 10:25:46.364748 17923 ts_tablet_manager.cc:1939] T 691d1699bfe2441f9b58bf63d144dc9f P 53e255590c734aa3950261d1443d9e27: Deleting consensus metadata
I20260925 10:25:46.365995 17922 ts_tablet_manager.cc:1918] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:46.366163 17922 log.cc:1196] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/691d1699bfe2441f9b58bf63d144dc9f
I20260925 10:25:46.367038 17407 catalog_manager.cc:4841] TS 53e255590c734aa3950261d1443d9e27 (127.12.248.67:32965): tablet 691d1699bfe2441f9b58bf63d144dc9f (table test-table [id=a8e8086d82664a39ace2b202c54e837b]) successfully deleted
I20260925 10:25:46.368446 17921 ts_tablet_manager.cc:1918] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:46.368613 17921 log.cc:1196] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/691d1699bfe2441f9b58bf63d144dc9f
Sep 25 10:25:46 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:46.369068 17921 ts_tablet_manager.cc:1939] T 691d1699bfe2441f9b58bf63d144dc9f P 9f1d0aaa4019451a852ce71b1d9cc243: Deleting consensus metadata
I20260925 10:25:46.370944 17408 catalog_manager.cc:4841] TS 9f1d0aaa4019451a852ce71b1d9cc243 (127.12.248.66:34797): tablet 691d1699bfe2441f9b58bf63d144dc9f (table test-table [id=a8e8086d82664a39ace2b202c54e837b]) successfully deleted
I20260925 10:25:46.371302 17922 ts_tablet_manager.cc:1939] T 691d1699bfe2441f9b58bf63d144dc9f P 7c9a5a0801b84868bb7e75fcfb5f024d: Deleting consensus metadata
I20260925 10:25:46.373034 17408 catalog_manager.cc:4841] TS 7c9a5a0801b84868bb7e75fcfb5f024d (127.12.248.65:37147): tablet 691d1699bfe2441f9b58bf63d144dc9f (table test-table [id=a8e8086d82664a39ace2b202c54e837b]) successfully deleted
I20260925 10:25:46.378268 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.332660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.12.248.65:37147)
0925 10:25:46.332839 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.332845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.348308 (+ 15463us) server_negotiation.cc:407] Connection header received
0925 10:25:46.348836 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.348843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.348943 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.349018 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:46.352904 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.354019 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.355915 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.356572 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.356671 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.370316 (+ 13645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:46.370333 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:46.370352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:46.370408 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:46.375067 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.375856 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.375866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.375870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.376058 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.376447 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.376454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.376459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.376924 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:46.377263 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.377583 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.377844 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:25:46.378923 17605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:46.469645 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17482
I20260925 10:25:46.493391 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17616
Sep 25 10:25:46 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:46.509302 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.494243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.12.248.66:34797)
0925 10:25:46.494430 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.494436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.494458 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:25:46.494624 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.494630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.494722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.494849 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:46.496032 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.497181 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.498504 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.499115 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.499258 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.503625 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:46.503641 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:46.503658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:46.503695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:46.506559 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.507227 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.507235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.507237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.507374 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.507748 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.507766 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.507770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.508116 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:46.508431 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.508735 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.508972 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:46.510030 17739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:46.600682 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17616
I20260925 10:25:46.618146 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17749
Sep 25 10:25:46 dist-test-slave-f9rl krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:46.635071 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:46.618975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.12.248.67:32965)
0925 10:25:46.619150 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:46.619158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:46.619207 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:25:46.619470 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:46.619474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:46.619540 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:46.619652 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:46.620924 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.621714 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.622968 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:46.623387 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:46.623538 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:46.628028 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:46.628044 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:46.628063 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:46.628125 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:46.632203 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.632876 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.632886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.632890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.633060 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:46.633466 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:46.633472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:46.633475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:46.633951 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:46.634282 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:46.634480 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 10:25:46.634700 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:25:46.635866 17872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:46.756423 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17749
I20260925 10:25:46.775319 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17390
I20260925 10:25:46.776467 17451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:46.887620 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17390
2026-09-25T10:25:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4299 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:25:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:46Z Disabled control of system clock
I20260925 10:25:46.955453 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38487
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:25:47.285159 17932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:47.285529 17932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:47.285631 17932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:47.292887 17932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:47.292994 17932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:47.293033 17932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:47.293084 17932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:47.293120 17932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:47.302496 17932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:47.303766 17932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:47.305532 17932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:47.317809 17940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:47.317726 17937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:47.317767 17938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:47.318565 17932 server_base.cc:1034] running on GCE node
I20260925 10:25:47.319375 17932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:47.320820 17932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:47.322058 17932 hybrid_clock.cc:648] HybridClock initialized: now 1790331947322017 us; error 51 us; skew 500 ppm
I20260925 10:25:47.322431 17932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:47.324667 17932 webserver.cc:460] Webserver started at http://127.12.248.126:46825/ using document root <none> and password file <none>
I20260925 10:25:47.325371 17932 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:47.325525 17932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:47.325905 17932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:47.328761 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "ce658004d2a3495d88758172c7136675"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "4ffdef5e4b902274deadb0bb79d82af6"
server_key_iv: "4b284fde7a5a1298a822800a748d60fe"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.329617 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "ce658004d2a3495d88758172c7136675"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "4ffdef5e4b902274deadb0bb79d82af6"
server_key_iv: "4b284fde7a5a1298a822800a748d60fe"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.334728 17932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:25:47.338100 17946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:47.339414 17932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:47.339617 17932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "ce658004d2a3495d88758172c7136675"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "4ffdef5e4b902274deadb0bb79d82af6"
server_key_iv: "4b284fde7a5a1298a822800a748d60fe"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.339843 17932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:47.345646 17932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:47.346410 17932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:47.346716 17932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:47.365181 17932 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:38487
I20260925 10:25:47.365200 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:38487 every 8 connection(s)
I20260925 10:25:47.366641 17932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:47.370430 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:47.373095 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 17932
I20260925 10:25:47.373327 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:47.374032 13281 external_mini_cluster.cc:1428] Setting key 65d7c57461ba085ef4879a9153f200dc
I20260925 10:25:47.384034 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:47.376323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.12.248.126:38487)
0925 10:25:47.377504 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:47.377520 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:47.377575 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:25:47.379809 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:47.379846 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:47.380460 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:47.380941 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:47.382032 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:47.382042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:47.382064 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:47.382082 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:47.382155 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:47.382176 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:47.382420 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:47.382540 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 10:25:47.382801 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":349,"threads_started":1}
I20260925 10:25:47.384553 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: Bootstrap starting.
I20260925 10:25:47.388054 17998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:47.389070 17998 log.cc:824] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:47.391870 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: No bootstrap required, opened a new log
I20260925 10:25:47.399144 17998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } }
I20260925 10:25:47.399542 17998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:47.399641 17998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce658004d2a3495d88758172c7136675, State: Initialized, Role: FOLLOWER
I20260925 10:25:47.400265 17998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } }
I20260925 10:25:47.400496 17998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:47.400650 17998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:47.400835 17998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:47.403280 17998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } }
I20260925 10:25:47.403824 17998 leader_election.cc:302] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce658004d2a3495d88758172c7136675; no voters:
I20260925 10:25:47.404413 17998 leader_election.cc:288] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:47.404574 18003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:47.405375 18003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [term 1 LEADER]: Becoming Leader. State: Replica: ce658004d2a3495d88758172c7136675, State: Running, Role: LEADER
I20260925 10:25:47.405951 18003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } }
I20260925 10:25:47.406610 17998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:47.409001 18005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce658004d2a3495d88758172c7136675. Latest consensus state: current_term: 1 leader_uuid: "ce658004d2a3495d88758172c7136675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } } }
I20260925 10:25:47.409192 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce658004d2a3495d88758172c7136675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce658004d2a3495d88758172c7136675" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38487 } } }
I20260925 10:25:47.409524 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:47.409617 18005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:47.415015 18012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:47.419180 18012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:47.427835 18012 catalog_manager.cc:1347] Generated new cluster ID: 15fa39ce5ccc461caa88755f29e8987d
I20260925 10:25:47.427963 18012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:47.455410 18012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:47.455855 18012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:47.466148 18012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: Generated new TSK 0
I20260925 10:25:47.467240 18012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:25:47.472306 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--builtin_ntp_servers=127.12.248.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:25:47.793929 18023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:47.794224 18023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:47.794315 18023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:47.801805 18023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:47.801908 18023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:47.802014 18023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:47.811511 18023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:47.812873 18023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:47.814621 18023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:47.827397 18028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:47.827307 18029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:47.827364 18031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:47.828406 18023 server_base.cc:1034] running on GCE node
I20260925 10:25:47.828965 18023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:47.830010 18023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:47.831243 18023 hybrid_clock.cc:648] HybridClock initialized: now 1790331947831180 us; error 81 us; skew 500 ppm
I20260925 10:25:47.831552 18023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:47.833848 18023 webserver.cc:460] Webserver started at http://127.12.248.65:44671/ using document root <none> and password file <none>
I20260925 10:25:47.834586 18023 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:47.834718 18023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:47.835052 18023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:47.837960 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "33025648149e44d1b55f98abb3520b34"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "383d18bd51ed94a0478899dae8ba0c53"
server_key_iv: "fc8484db03d4c958c7c3a2aa5502bd0d"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.838764 18023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "33025648149e44d1b55f98abb3520b34"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "383d18bd51ed94a0478899dae8ba0c53"
server_key_iv: "fc8484db03d4c958c7c3a2aa5502bd0d"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.844118 18023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:47.847622 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:47.848965 18023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:47.849102 18023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "33025648149e44d1b55f98abb3520b34"
format_stamp: "Formatted at 2026-09-25 10:25:47 on dist-test-slave-f9rl"
server_key: "383d18bd51ed94a0478899dae8ba0c53"
server_key_iv: "fc8484db03d4c958c7c3a2aa5502bd0d"
server_key_version: "encryptionkey@0"
I20260925 10:25:47.849305 18023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:47.855209 18023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:47.855963 18023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:47.856189 18023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:47.857074 18023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:47.858505 18023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:47.858592 18023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:47.858670 18023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:47.858724 18023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:47.891894 18023 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36491
I20260925 10:25:47.891942 18150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36491 every 8 connection(s)
I20260925 10:25:47.893234 18023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:47.901321 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18023
I20260925 10:25:47.901556 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:47.901751 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:47.895941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:36077 (local address 127.12.248.126:38487)
0925 10:25:47.896394 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:47.896399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:47.897693 (+ 1294us) server_negotiation.cc:407] Connection header received
0925 10:25:47.899428 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:47.899434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:47.899513 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:47.899634 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:47.900912 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:47.900916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:47.900919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:47.900937 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:47.900978 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:47.900989 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:47.901092 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:47.901342 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:25:47.901391 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260925 10:25:47.902403 13281 external_mini_cluster.cc:1428] Setting key 121732977bc7be8a6da2b3f0c2902679
I20260925 10:25:47.903007 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:47.896569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38487 (local address 127.12.248.65:36077)
0925 10:25:47.897532 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:47.897575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:47.899136 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:47.899870 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:47.899878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:47.899969 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:47.900589 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:47.900592 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:47.900629 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:47.900713 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:47.901097 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:47.901110 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:47.901117 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:25:47.901429 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 10:25:47.901733 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260925 10:25:47.904539 18151 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38487
I20260925 10:25:47.904868 18151 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:47.905558 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--builtin_ntp_servers=127.12.248.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:25:47.905627 18151 heartbeater.cc:511] Master 127.12.248.126:38487 requested a full tablet report, sending...
I20260925 10:25:47.908071 17963 ts_manager.cc:194] Registered new tserver with Master: 33025648149e44d1b55f98abb3520b34 (127.12.248.65:36491)
W20260925 10:25:48.255904 18155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:48.256242 18155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:48.256402 18155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:48.263796 18155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:48.263943 18155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:48.264137 18155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:48.273535 18155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:48.274906 18155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:48.276713 18155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:48.289829 18160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:48.289768 18163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:48.289799 18161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:48.290660 18155 server_base.cc:1034] running on GCE node
I20260925 10:25:48.291193 18155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:48.292050 18155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:48.293273 18155 hybrid_clock.cc:648] HybridClock initialized: now 1790331948293230 us; error 38 us; skew 500 ppm
I20260925 10:25:48.293629 18155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:48.296017 18155 webserver.cc:460] Webserver started at http://127.12.248.66:37197/ using document root <none> and password file <none>
I20260925 10:25:48.296669 18155 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:48.296766 18155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:48.297039 18155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:48.299998 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "09807b6d8f1d465ebc9f7311454f8236"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "7a5de1fd8458dc818f7eafd87865e724"
server_key_iv: "c71366cefb99705248970d3d5660e816"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.300752 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "09807b6d8f1d465ebc9f7311454f8236"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "7a5de1fd8458dc818f7eafd87865e724"
server_key_iv: "c71366cefb99705248970d3d5660e816"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.306241 18155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:25:48.309862 18169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.311370 18155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:25:48.311504 18155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "09807b6d8f1d465ebc9f7311454f8236"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "7a5de1fd8458dc818f7eafd87865e724"
server_key_iv: "c71366cefb99705248970d3d5660e816"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.311686 18155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:48.317253 18155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:48.318053 18155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:48.318269 18155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:48.319103 18155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:48.320403 18155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:48.320487 18155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.320566 18155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:48.320621 18155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.354604 18155 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:43591
I20260925 10:25:48.354653 18282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:43591 every 8 connection(s)
I20260925 10:25:48.355926 18155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:48.357286 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18155
I20260925 10:25:48.357510 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:48.358026 13281 external_mini_cluster.cc:1428] Setting key 5077cbd7ae72f6aba55485f2524fcd0e
I20260925 10:25:48.360868 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--builtin_ntp_servers=127.12.248.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:25:48.364861 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.358717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:35821 (local address 127.12.248.126:38487)
0925 10:25:48.358948 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.358954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.360591 (+ 1637us) server_negotiation.cc:407] Connection header received
0925 10:25:48.362445 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.362453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.362535 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.362647 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.363965 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.363974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.363981 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.364004 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.364046 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.364064 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.364191 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.364355 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.364425 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:25:48.365903 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.359344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38487 (local address 127.12.248.66:35821)
0925 10:25:48.360411 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.360456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.362166 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.362861 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.362869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.362961 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.363654 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.363656 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.363694 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.363789 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.364163 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.364170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.364176 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.364428 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.364689 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":354,"threads_started":1}
I20260925 10:25:48.367355 18283 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38487
I20260925 10:25:48.367659 18283 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:48.368327 18283 heartbeater.cc:511] Master 127.12.248.126:38487 requested a full tablet report, sending...
I20260925 10:25:48.369812 17963 ts_manager.cc:194] Registered new tserver with Master: 09807b6d8f1d465ebc9f7311454f8236 (127.12.248.66:43591)
W20260925 10:25:48.693960 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:48.694247 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:48.694337 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:48.701606 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:48.701717 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:48.701840 18286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:48.711470 18286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:48.712853 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:48.714872 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:48.728411 18291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:48.728379 18294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:48.728370 18292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:48.729090 18286 server_base.cc:1034] running on GCE node
I20260925 10:25:48.729794 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:48.730940 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:48.732218 18286 hybrid_clock.cc:648] HybridClock initialized: now 1790331948732154 us; error 52 us; skew 500 ppm
I20260925 10:25:48.732554 18286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:48.734921 18286 webserver.cc:460] Webserver started at http://127.12.248.67:43469/ using document root <none> and password file <none>
I20260925 10:25:48.735641 18286 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:48.735773 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:48.736107 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:48.738971 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "53075877b33349f8aa7ab5e30c658953"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "9e7cc4ee33fb804c0667d59d393e48f7"
server_key_iv: "bd89a9b73723859b617d0128a3e7e3b3"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.739784 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "53075877b33349f8aa7ab5e30c658953"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "9e7cc4ee33fb804c0667d59d393e48f7"
server_key_iv: "bd89a9b73723859b617d0128a3e7e3b3"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.745038 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:25:48.748489 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.749888 18286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:48.750064 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "53075877b33349f8aa7ab5e30c658953"
format_stamp: "Formatted at 2026-09-25 10:25:48 on dist-test-slave-f9rl"
server_key: "9e7cc4ee33fb804c0667d59d393e48f7"
server_key_iv: "bd89a9b73723859b617d0128a3e7e3b3"
server_key_version: "encryptionkey@0"
I20260925 10:25:48.750301 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:48.756323 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:48.757103 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:48.757385 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:48.758343 18286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:48.759727 18286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:48.759847 18286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.759985 18286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:48.760074 18286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:48.793226 18286 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:45389
I20260925 10:25:48.793283 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:45389 every 8 connection(s)
I20260925 10:25:48.794623 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:48.795311 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18286
I20260925 10:25:48.795487 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:48.796085 13281 external_mini_cluster.cc:1428] Setting key b456eec419d1aa662c4dffb7131462dd
I20260925 10:25:48.803181 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.797541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:40501 (local address 127.12.248.126:38487)
0925 10:25:48.797745 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.797753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.799448 (+ 1695us) server_negotiation.cc:407] Connection header received
0925 10:25:48.801121 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.801125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.801176 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.801261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.802589 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.802593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.802596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.802611 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.802639 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.802650 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.802749 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.802904 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.802951 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:48.804100 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.798192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38487 (local address 127.12.248.67:40501)
0925 10:25:48.799286 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.799337 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.800887 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.801522 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.801529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.801623 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.802324 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.802326 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.802363 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.802447 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.802735 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.802742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.802747 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.802953 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.803150 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
I20260925 10:25:48.805390 18414 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38487
I20260925 10:25:48.805704 18414 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:48.806313 18414 heartbeater.cc:511] Master 127.12.248.126:38487 requested a full tablet report, sending...
I20260925 10:25:48.807667 17963 ts_manager.cc:194] Registered new tserver with Master: 53075877b33349f8aa7ab5e30c658953 (127.12.248.67:45389)
I20260925 10:25:48.811620 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:25:48.818487 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.816196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.12.248.126:38487)
0925 10:25:48.816367 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.816371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.816870 (+ 499us) server_negotiation.cc:407] Connection header received
0925 10:25:48.816989 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.816992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.817039 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.817118 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.817997 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.818001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.818004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.818016 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.818044 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.818054 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.818149 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.818215 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.818269 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:25:48.822230 17963 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:25:48.829056 17963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:25:48.847694 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.841482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.12.248.65:36491)
0925 10:25:48.842577 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.842589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.844593 (+ 2004us) server_negotiation.cc:407] Connection header received
0925 10:25:48.844900 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.844908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.845132 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.845251 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.846701 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.846709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.846723 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.846753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.846825 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.846848 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.846961 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.847244 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.847350 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":797,"threads_started":1}
I20260925 10:25:48.847764 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.840935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.12.248.67:45389)
0925 10:25:48.841959 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.841972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.843590 (+ 1618us) server_negotiation.cc:407] Connection header received
0925 10:25:48.843892 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.843903 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.844100 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.844196 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.846398 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.846407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.846421 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.846439 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.846500 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.846522 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.846922 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.847248 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.847351 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":1057,"threads_started":1}
I20260925 10:25:48.848400 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.841845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.12.248.66:43591)
0925 10:25:48.844069 (+ 2224us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.844090 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.845880 (+ 1790us) server_negotiation.cc:407] Connection header received
0925 10:25:48.845957 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.845964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.846159 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.846311 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.847213 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.847221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.847232 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.847250 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.847310 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.847328 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.847477 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.847979 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.848069 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2021,"thread_start_us":1875,"threads_started":1}
I20260925 10:25:48.848677 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.840921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36491 (local address 127.0.0.1:50628)
0925 10:25:48.844484 (+ 3563us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.844501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.844719 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.845606 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.845610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.845633 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.846380 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.846383 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.846398 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.846451 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.847052 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.847056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.847061 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.847244 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.848135 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3434,"thread_start_us":3325,"threads_started":1}
I20260925 10:25:48.849169 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.840898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:45389 (local address 127.0.0.1:45046)
0925 10:25:48.843408 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.843466 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.843719 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.844746 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.844756 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.844847 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.845800 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.845806 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.845843 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.845934 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.847026 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.847037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.847044 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.848337 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.848392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2190,"thread_start_us":4401,"threads_started":1}
I20260925 10:25:48.849287 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.841322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:43591 (local address 127.0.0.1:45978)
0925 10:25:48.845584 (+ 4262us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.845602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.845713 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.846349 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.846355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.846377 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.846999 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.847004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.847018 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.847055 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.847532 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.847539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.847545 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.847687 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.848577 (+ 890us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4097,"thread_start_us":1562,"threads_started":1}
I20260925 10:25:48.852602 18085 tablet_service.cc:1467] Processing CreateTablet for tablet f96e7cbd54744910befed2b16dcadc2f (DEFAULT_TABLE table=test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:48.852968 18217 tablet_service.cc:1467] Processing CreateTablet for tablet f96e7cbd54744910befed2b16dcadc2f (DEFAULT_TABLE table=test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:48.853531 18348 tablet_service.cc:1467] Processing CreateTablet for tablet f96e7cbd54744910befed2b16dcadc2f (DEFAULT_TABLE table=test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:25:48.854349 18085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96e7cbd54744910befed2b16dcadc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:48.854593 18217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96e7cbd54744910befed2b16dcadc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:48.854974 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f96e7cbd54744910befed2b16dcadc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:48.866503 18432 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Bootstrap starting.
I20260925 10:25:48.870564 18433 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Bootstrap starting.
I20260925 10:25:48.870781 18432 tablet_bootstrap.cc:654] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:48.872143 18432 log.cc:824] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:48.873648 18434 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Bootstrap starting.
I20260925 10:25:48.874670 18433 tablet_bootstrap.cc:654] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:48.875607 18432 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: No bootstrap required, opened a new log
I20260925 10:25:48.875839 18432 ts_tablet_manager.cc:1397] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 10:25:48.875898 18433 log.cc:824] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:48.877161 18434 tablet_bootstrap.cc:654] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:48.878383 18434 log.cc:824] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:48.879218 18433 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: No bootstrap required, opened a new log
I20260925 10:25:48.879490 18433 ts_tablet_manager.cc:1397] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:25:48.881567 18434 tablet_bootstrap.cc:492] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: No bootstrap required, opened a new log
I20260925 10:25:48.881901 18434 ts_tablet_manager.cc:1397] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 10:25:48.885062 18432 raft_consensus.cc:348] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.885497 18432 raft_consensus.cc:374] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:48.885608 18432 raft_consensus.cc:729] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33025648149e44d1b55f98abb3520b34, State: Initialized, Role: FOLLOWER
I20260925 10:25:48.886315 18432 consensus_queue.cc:260] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.887522 18151 heartbeater.cc:503] Master 127.12.248.126:38487 was elected leader, sending a full tablet report...
I20260925 10:25:48.888285 18432 ts_tablet_manager.cc:1428] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 10:25:48.890136 18433 raft_consensus.cc:348] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.890594 18433 raft_consensus.cc:374] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:48.890681 18433 raft_consensus.cc:729] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09807b6d8f1d465ebc9f7311454f8236, State: Initialized, Role: FOLLOWER
I20260925 10:25:48.891426 18433 consensus_queue.cc:260] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.892540 18434 raft_consensus.cc:348] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.892969 18434 raft_consensus.cc:374] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:48.893070 18434 raft_consensus.cc:729] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53075877b33349f8aa7ab5e30c658953, State: Initialized, Role: FOLLOWER
I20260925 10:25:48.893287 18433 ts_tablet_manager.cc:1428] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 10:25:48.893774 18283 heartbeater.cc:503] Master 127.12.248.126:38487 was elected leader, sending a full tablet report...
I20260925 10:25:48.893664 18434 consensus_queue.cc:260] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.894775 18414 heartbeater.cc:503] Master 127.12.248.126:38487 was elected leader, sending a full tablet report...
I20260925 10:25:48.895375 18434 ts_tablet_manager.cc:1428] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260925 10:25:48.896099 18152 tablet.cc:2369] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:25:48.975957 18439 raft_consensus.cc:482] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:25:48.976341 18439 raft_consensus.cc:504] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.978567 18439 leader_election.cc:288] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53075877b33349f8aa7ab5e30c658953 (127.12.248.67:45389), 33025648149e44d1b55f98abb3520b34 (127.12.248.65:36491)
I20260925 10:25:48.982641 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.979520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:59839 (local address 127.12.248.65:36491)
0925 10:25:48.979740 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.979749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.980518 (+ 769us) server_negotiation.cc:407] Connection header received
0925 10:25:48.980836 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.980840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.980889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.980996 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.981981 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.981987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.981991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.982010 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.982049 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.982060 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.982180 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.982280 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.982346 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:25:48.982771 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.979098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:56253 (local address 127.12.248.67:45389)
0925 10:25:48.979325 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:48.979333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:48.980762 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 10:25:48.980905 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:48.980910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:48.980971 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:48.981077 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:48.982142 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:48.982147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:48.982152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:48.982169 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:48.982208 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:48.982219 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:48.982353 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:48.982442 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:25:48.982502 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:25:48.982961 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.978929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36491 (local address 127.12.248.66:59839)
0925 10:25:48.980404 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.980424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.980650 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.981019 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.981024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.981048 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.981777 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.981780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.981794 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.981831 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.982152 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.982156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.982160 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.982314 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.982362 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":1372,"threads_started":1}
I20260925 10:25:48.983359 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:48.979114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:45389 (local address 127.12.248.66:56253)
0925 10:25:48.980652 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:48.980668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:48.980799 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:48.981102 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:48.981107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:48.981135 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:48.981957 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:25:48.981960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:48.981972 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:25:48.982004 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:48.982321 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:48.982325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:48.982331 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:25:48.982491 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 10:25:48.982536 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"thread_start_us":821,"threads_started":1}
I20260925 10:25:48.983912 18105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96e7cbd54744910befed2b16dcadc2f" candidate_uuid: "09807b6d8f1d465ebc9f7311454f8236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33025648149e44d1b55f98abb3520b34" is_pre_election: true
I20260925 10:25:48.984341 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96e7cbd54744910befed2b16dcadc2f" candidate_uuid: "09807b6d8f1d465ebc9f7311454f8236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53075877b33349f8aa7ab5e30c658953" is_pre_election: true
I20260925 10:25:48.984536 18105 raft_consensus.cc:2454] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09807b6d8f1d465ebc9f7311454f8236 in term 0.
I20260925 10:25:48.984887 18368 raft_consensus.cc:2454] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09807b6d8f1d465ebc9f7311454f8236 in term 0.
I20260925 10:25:48.985240 18172 leader_election.cc:302] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09807b6d8f1d465ebc9f7311454f8236, 33025648149e44d1b55f98abb3520b34; no voters:
I20260925 10:25:48.985682 18439 raft_consensus.cc:2781] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:25:48.985845 18439 raft_consensus.cc:482] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:25:48.985949 18439 raft_consensus.cc:3037] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:48.988091 18439 raft_consensus.cc:504] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.988665 18439 leader_election.cc:288] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [CANDIDATE]: Term 1 election: Requested vote from peers 53075877b33349f8aa7ab5e30c658953 (127.12.248.67:45389), 33025648149e44d1b55f98abb3520b34 (127.12.248.65:36491)
I20260925 10:25:48.989018 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96e7cbd54744910befed2b16dcadc2f" candidate_uuid: "09807b6d8f1d465ebc9f7311454f8236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53075877b33349f8aa7ab5e30c658953"
I20260925 10:25:48.989220 18368 raft_consensus.cc:3037] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:48.989140 18105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f96e7cbd54744910befed2b16dcadc2f" candidate_uuid: "09807b6d8f1d465ebc9f7311454f8236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33025648149e44d1b55f98abb3520b34"
I20260925 10:25:48.989400 18105 raft_consensus.cc:3037] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:48.991524 18368 raft_consensus.cc:2454] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09807b6d8f1d465ebc9f7311454f8236 in term 1.
I20260925 10:25:48.991750 18105 raft_consensus.cc:2454] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09807b6d8f1d465ebc9f7311454f8236 in term 1.
I20260925 10:25:48.991918 18170 leader_election.cc:302] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09807b6d8f1d465ebc9f7311454f8236, 53075877b33349f8aa7ab5e30c658953; no voters:
I20260925 10:25:48.992244 18439 raft_consensus.cc:2781] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:48.992704 18439 raft_consensus.cc:686] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 1 LEADER]: Becoming Leader. State: Replica: 09807b6d8f1d465ebc9f7311454f8236, State: Running, Role: LEADER
I20260925 10:25:48.993153 18439 consensus_queue.cc:237] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } }
I20260925 10:25:48.997620 17962 catalog_manager.cc:5495] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09807b6d8f1d465ebc9f7311454f8236 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "09807b6d8f1d465ebc9f7311454f8236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09807b6d8f1d465ebc9f7311454f8236" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 43591 } health_report { overall_health: HEALTHY } } }
I20260925 10:25:49.019762 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:49.017750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.12.248.66:43591)
0925 10:25:49.017916 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:49.017920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:49.018067 (+ 147us) server_negotiation.cc:407] Connection header received
0925 10:25:49.018253 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:49.018256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:49.018351 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:49.018446 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:49.019269 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:49.019273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:49.019276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:49.019291 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:49.019320 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:49.019331 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:49.019439 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:49.019526 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:25:49.019568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:25:49.027755 18105 raft_consensus.cc:1261] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 1 FOLLOWER]: Refusing update from remote peer 09807b6d8f1d465ebc9f7311454f8236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:49.027767 18368 raft_consensus.cc:1261] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 1 FOLLOWER]: Refusing update from remote peer 09807b6d8f1d465ebc9f7311454f8236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:25:49.028698 18443 consensus_queue.cc:1035] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33025648149e44d1b55f98abb3520b34" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:49.029030 18439 consensus_queue.cc:1035] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53075877b33349f8aa7ab5e30c658953" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:25:49.040104 18446 mvcc.cc:205] Tried to move back new op lower bound from 7333199663207981056 to 7333199663080091648. Current Snapshot: MvccSnapshot[applied={T|T < 7333199663207981056}]
I20260925 10:25:49.042580 18447 mvcc.cc:205] Tried to move back new op lower bound from 7333199663207981056 to 7333199663080091648. Current Snapshot: MvccSnapshot[applied={T|T < 7333199663207981056}]
I20260925 10:25:49.056878 17963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:49.057156 17963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:25:49.060573 17963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce658004d2a3495d88758172c7136675: Sending DeleteTablet for 3 replicas of tablet f96e7cbd54744910befed2b16dcadc2f
I20260925 10:25:49.061734 18085 tablet_service.cc:1514] Processing DeleteTablet for tablet f96e7cbd54744910befed2b16dcadc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:49 UTC) from {username='slave'} at 127.0.0.1:50628
I20260925 10:25:49.061805 18217 tablet_service.cc:1514] Processing DeleteTablet for tablet f96e7cbd54744910befed2b16dcadc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:49 UTC) from {username='slave'} at 127.0.0.1:45978
I20260925 10:25:49.062644 18455 tablet_replica.cc:330] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: stopping tablet replica
I20260925 10:25:49.062769 18348 tablet_service.cc:1514] Processing DeleteTablet for tablet f96e7cbd54744910befed2b16dcadc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:25:49 UTC) from {username='slave'} at 127.0.0.1:45046
I20260925 10:25:49.063081 18455 raft_consensus.cc:2229] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:49.063674 18455 raft_consensus.cc:2258] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:49.065966 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18023
I20260925 10:25:49.067026 18455 ts_tablet_manager.cc:1905] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:49.076998 18455 ts_tablet_manager.cc:1918] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:49.077240 18455 log.cc:1196] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/f96e7cbd54744910befed2b16dcadc2f
I20260925 10:25:49.077857 18455 ts_tablet_manager.cc:1939] T f96e7cbd54744910befed2b16dcadc2f P 33025648149e44d1b55f98abb3520b34: Deleting consensus metadata
I20260925 10:25:49.078001 18457 tablet_replica.cc:330] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: stopping tablet replica
I20260925 10:25:49.078475 18457 raft_consensus.cc:2229] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:25:49.079002 18457 raft_consensus.cc:2258] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:49.079849 17949 catalog_manager.cc:4841] TS 33025648149e44d1b55f98abb3520b34 (127.12.248.65:36491): tablet f96e7cbd54744910befed2b16dcadc2f (table test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]) successfully deleted
I20260925 10:25:49.081326 18456 tablet_replica.cc:330] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: stopping tablet replica
I20260925 10:25:49.081787 18456 raft_consensus.cc:2229] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:25:49.082671 18456 raft_consensus.cc:2258] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:25:49.083289 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:49.068770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.12.248.65:36491)
0925 10:25:49.068943 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:49.068950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:49.080594 (+ 11644us) server_negotiation.cc:407] Connection header received
0925 10:25:49.080822 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:49.080830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:49.080944 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:49.081052 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:49.082308 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:49.082316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:49.082321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:49.082404 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:49.082446 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:49.082464 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:49.082617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:49.082912 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 10:25:49.082977 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:25:49.084231 18457 ts_tablet_manager.cc:1905] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:49.084570 18146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:49.086529 18456 ts_tablet_manager.cc:1905] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:25:49.092789 18457 ts_tablet_manager.cc:1918] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:49.092970 18457 log.cc:1196] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/f96e7cbd54744910befed2b16dcadc2f
I20260925 10:25:49.093654 18457 ts_tablet_manager.cc:1939] T f96e7cbd54744910befed2b16dcadc2f P 53075877b33349f8aa7ab5e30c658953: Deleting consensus metadata
I20260925 10:25:49.096119 17947 catalog_manager.cc:4841] TS 53075877b33349f8aa7ab5e30c658953 (127.12.248.67:45389): tablet f96e7cbd54744910befed2b16dcadc2f (table test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]) successfully deleted
I20260925 10:25:49.098259 18456 ts_tablet_manager.cc:1918] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:25:49.098448 18456 log.cc:1196] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/f96e7cbd54744910befed2b16dcadc2f
I20260925 10:25:49.099036 18456 ts_tablet_manager.cc:1939] T f96e7cbd54744910befed2b16dcadc2f P 09807b6d8f1d465ebc9f7311454f8236: Deleting consensus metadata
I20260925 10:25:49.100873 17948 catalog_manager.cc:4841] TS 09807b6d8f1d465ebc9f7311454f8236 (127.12.248.66:43591): tablet f96e7cbd54744910befed2b16dcadc2f (table test-table [id=8f376a6ccb9f4a76b0c8642b087ea0f7]) successfully deleted
I20260925 10:25:49.185061 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18023
I20260925 10:25:49.207988 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18155
I20260925 10:25:49.211072 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:49.208834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.12.248.66:43591)
0925 10:25:49.209024 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:49.209031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:49.209053 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:25:49.209205 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:49.209211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:49.209274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:49.209392 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:49.210368 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:49.210375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:49.210381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:49.210400 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:49.210447 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:49.210465 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:49.210600 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:49.210704 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 10:25:49.210760 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:25:49.211843 18278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:49.321560 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18155
I20260925 10:25:49.341869 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18286
I20260925 10:25:49.344745 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:49.342643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.12.248.67:45389)
0925 10:25:49.342835 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:49.342840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:49.342857 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:25:49.343062 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:49.343066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:49.343137 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:49.343290 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:49.344186 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:49.344190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:49.344193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:49.344205 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:49.344232 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:49.344243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:49.344368 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:49.344432 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:25:49.344513 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:25:49.345481 18409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:49.434688 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18286
I20260925 10:25:49.450985 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 17932
I20260925 10:25:49.452180 17993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:49.558911 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 17932
2026-09-25T10:25:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2669 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:25:49.599884 13281 mini_oidc.cc:152] Starting JWKS server
I20260925 10:25:49.602602 13281 webserver.cc:460] Webserver started at https://127.0.0.1:42771/ using document root <none> and password file <none>
I20260925 10:25:49.602916 13281 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:25:49.603492 13281 webserver.cc:460] Webserver started at http://127.0.0.1:46863/ using document root <none> and password file <none>
2026-09-25T10:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:49Z Disabled control of system clock
I20260925 10:25:49.629541 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36047
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:45065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:36047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/testchainca.cert with env {}
W20260925 10:25:49.949733 18468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:49.950085 18468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:49.950258 18468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:49.957577 18468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:49.957716 18468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:49.957784 18468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:49.957870 18468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:49.957960 18468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:25:49.958050 18468 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:25:49.958110 18468 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42771/jwks.json
I20260925 10:25:49.968168 18468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:45065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:36047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:49.969580 18468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:49.970890 18468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:49.983495 18474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:49.983474 18473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:49.984004 18476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:49.984740 18468 server_base.cc:1034] running on GCE node
I20260925 10:25:49.985613 18468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:49.987040 18468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:49.988276 18468 hybrid_clock.cc:648] HybridClock initialized: now 1790331949988243 us; error 46 us; skew 500 ppm
I20260925 10:25:49.988632 18468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:49.990868 18468 webserver.cc:460] Webserver started at http://127.12.248.126:39597/ using document root <none> and password file <none>
I20260925 10:25:49.991600 18468 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:49.991740 18468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:49.992050 18468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:49.995008 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "46c3dc6d7de849bd97c5538e42833d3f"
format_stamp: "Formatted at 2026-09-25 10:25:49 on dist-test-slave-f9rl"
server_key: "e184b277286c99a57f065c836efd77f7"
server_key_iv: "1c41c170e7fdb3de88c8323a954d4f8f"
server_key_version: "encryptionkey@0"
I20260925 10:25:49.995810 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "46c3dc6d7de849bd97c5538e42833d3f"
format_stamp: "Formatted at 2026-09-25 10:25:49 on dist-test-slave-f9rl"
server_key: "e184b277286c99a57f065c836efd77f7"
server_key_iv: "1c41c170e7fdb3de88c8323a954d4f8f"
server_key_version: "encryptionkey@0"
I20260925 10:25:50.000896 18468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:25:50.004391 18482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:50.005859 18468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:25:50.006047 18468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "46c3dc6d7de849bd97c5538e42833d3f"
format_stamp: "Formatted at 2026-09-25 10:25:49 on dist-test-slave-f9rl"
server_key: "e184b277286c99a57f065c836efd77f7"
server_key_iv: "1c41c170e7fdb3de88c8323a954d4f8f"
server_key_version: "encryptionkey@0"
I20260925 10:25:50.006261 18468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:50.051717 18468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:50.052651 18468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:50.053000 18468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:50.072845 18468 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:36047
I20260925 10:25:50.072901 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:36047 every 8 connection(s)
I20260925 10:25:50.074329 18468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:50.077073 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18468
I20260925 10:25:50.077322 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:50.077986 13281 external_mini_cluster.cc:1428] Setting key cbae985d0246b38f552c76a944d75ddd
I20260925 10:25:50.079490 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:50.094866 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f: Bootstrap starting.
I20260925 10:25:50.095461 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:50.080067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.12.248.126:36047)
0925 10:25:50.081254 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:50.081269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:50.081324 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:25:50.083242 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:50.083272 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:50.083932 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:50.084397 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:50.085142 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.087311 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.089580 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.090665 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.090887 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.092314 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:50.092324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:50.092347 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:50.093077 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:50.093191 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:50.093220 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:50.093341 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:50.093779 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 10:25:50.094115 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":398,"threads_started":1}
I20260925 10:25:50.098922 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:50.100255 18534 log.cc:824] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:50.107636 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:25:50.108242 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f: No bootstrap required, opened a new log
I20260925 10:25:50.116355 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } }
I20260925 10:25:50.116806 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:50.116938 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c3dc6d7de849bd97c5538e42833d3f, State: Initialized, Role: FOLLOWER
I20260925 10:25:50.117694 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } }
I20260925 10:25:50.117949 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:50.118129 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:50.118328 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:50.120651 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } }
I20260925 10:25:50.121183 18534 leader_election.cc:302] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c3dc6d7de849bd97c5538e42833d3f; no voters:
I20260925 10:25:50.121857 18534 leader_election.cc:288] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:50.122082 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:50.122931 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [term 1 LEADER]: Becoming Leader. State: Replica: 46c3dc6d7de849bd97c5538e42833d3f, State: Running, Role: LEADER
I20260925 10:25:50.123602 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } }
I20260925 10:25:50.124261 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:50.127182 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c3dc6d7de849bd97c5538e42833d3f. Latest consensus state: current_term: 1 leader_uuid: "46c3dc6d7de849bd97c5538e42833d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } } }
I20260925 10:25:50.127156 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c3dc6d7de849bd97c5538e42833d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c3dc6d7de849bd97c5538e42833d3f" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36047 } } }
I20260925 10:25:50.127663 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:50.127663 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:50.129544 18545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:50.135054 18545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:50.154381 18545 catalog_manager.cc:1347] Generated new cluster ID: 60d2cd550a704a3f972d58fbb3b9765e
I20260925 10:25:50.154513 18545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:50.184785 18545 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:50.186264 18545 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:50.199069 18545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46c3dc6d7de849bd97c5538e42833d3f: Generated new TSK 0
I20260925 10:25:50.200043 18545 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:25:50.247661 18564 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:25:50.247876 18564 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:25:50.249094 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:50.228142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.12.248.126:36047)
0925 10:25:50.228314 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:50.228318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:50.228890 (+ 572us) server_negotiation.cc:407] Connection header received
0925 10:25:50.229312 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:50.229322 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:50.229398 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:50.229607 (+ 209us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:25:50.230159 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.230947 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.232368 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.232853 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.232990 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.233543 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:25:50.248342 (+ 14799us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:25:50.248743 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 10:25:50.249228 18537 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 10:25:50.250627 18563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:50.228138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:36047 (local address 127.0.0.1:59880)
0925 10:25:50.228777 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:50.228797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:50.228995 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:50.229645 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:50.229651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:50.229664 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:25:50.230001 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:50.230009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.231137 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.231141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:50.232225 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:50.232236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.233020 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.233027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:50.233175 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.233231 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:25:50.248653 (+ 15422us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:25:50.249011 (+ 358us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.248.126:36047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":348,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.248.126:36047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:25:50.252640 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18468
I20260925 10:25:50.253772 18529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:50.350346 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18468
2026-09-25T10:25:50Z chronyd exiting
I20260925 10:25:50.402074 13281 test_util.cc:175] -----------------------------------------------
I20260925 10:25:50.402209 13281 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331906218802-13281-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (806 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:25:50.406026 13281 mini_oidc.cc:152] Starting JWKS server
I20260925 10:25:50.407871 13281 webserver.cc:460] Webserver started at https://127.0.0.1:43121/ using document root <none> and password file <none>
I20260925 10:25:50.408106 13281 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:25:50.408746 13281 webserver.cc:460] Webserver started at http://127.0.0.1:39801/ using document root <none> and password file <none>
2026-09-25T10:25:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:50Z Disabled control of system clock
I20260925 10:25:50.434587 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35929
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:35929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43121/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/testchainca.cert with env {}
W20260925 10:25:50.758157 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:50.758452 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:50.758646 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:50.766278 18571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:50.766420 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:50.766490 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:50.766600 18571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:50.766681 18571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:25:50.766783 18571 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:25:50.766842 18571 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43121/jwks.json
I20260925 10:25:50.777159 18571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:35929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43121/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:50.778621 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:50.780014 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:50.792337 18580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:50.792390 18577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:50.792409 18578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:50.793344 18571 server_base.cc:1034] running on GCE node
I20260925 10:25:50.794149 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:50.795495 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:50.796695 18571 hybrid_clock.cc:648] HybridClock initialized: now 1790331950796627 us; error 65 us; skew 500 ppm
I20260925 10:25:50.797065 18571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:50.804399 18571 webserver.cc:460] Webserver started at http://127.12.248.126:43203/ using document root <none> and password file <none>
I20260925 10:25:50.805140 18571 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:50.805243 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:50.805692 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:50.808619 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "86774cbe6bfa4a2ab69a2d752574e46c"
format_stamp: "Formatted at 2026-09-25 10:25:50 on dist-test-slave-f9rl"
server_key: "1c1f93649d2f8691744971b98080f053"
server_key_iv: "7044e52c4ca6b2ef0e2c0745334e0ac1"
server_key_version: "encryptionkey@0"
I20260925 10:25:50.809510 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "86774cbe6bfa4a2ab69a2d752574e46c"
format_stamp: "Formatted at 2026-09-25 10:25:50 on dist-test-slave-f9rl"
server_key: "1c1f93649d2f8691744971b98080f053"
server_key_iv: "7044e52c4ca6b2ef0e2c0745334e0ac1"
server_key_version: "encryptionkey@0"
I20260925 10:25:50.814742 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:25:50.818509 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:50.819924 18571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:50.820109 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "86774cbe6bfa4a2ab69a2d752574e46c"
format_stamp: "Formatted at 2026-09-25 10:25:50 on dist-test-slave-f9rl"
server_key: "1c1f93649d2f8691744971b98080f053"
server_key_iv: "7044e52c4ca6b2ef0e2c0745334e0ac1"
server_key_version: "encryptionkey@0"
I20260925 10:25:50.820343 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:50.849109 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:50.850144 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:50.850493 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:50.869212 18571 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:35929
I20260925 10:25:50.869310 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:35929 every 8 connection(s)
I20260925 10:25:50.870805 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:50.872488 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18571
I20260925 10:25:50.872691 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:50.873566 13281 external_mini_cluster.cc:1428] Setting key 3635b94eb705acbb5e635b93aaaada79
I20260925 10:25:50.875895 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:50.888172 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:50.875587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.12.248.126:35929)
0925 10:25:50.876618 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:50.876630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:50.876673 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:25:50.878515 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:50.878538 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:50.879060 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:50.879422 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:50.880038 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.881410 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.883294 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.883991 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.884131 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.885543 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:50.885553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:50.885568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:50.886263 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:50.886350 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:50.886372 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:50.886475 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:50.886716 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:25:50.887109 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":379,"threads_started":1}
I20260925 10:25:50.889333 18638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c: Bootstrap starting.
I20260925 10:25:50.892339 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:25:50.892890 18638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:50.893980 18638 log.cc:824] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:50.897176 18638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c: No bootstrap required, opened a new log
I20260925 10:25:50.904826 18638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } }
I20260925 10:25:50.905298 18638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:50.905462 18638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86774cbe6bfa4a2ab69a2d752574e46c, State: Initialized, Role: FOLLOWER
I20260925 10:25:50.906152 18638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } }
I20260925 10:25:50.906391 18638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:50.906524 18638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:50.906714 18638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:50.908993 18638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } }
I20260925 10:25:50.909606 18638 leader_election.cc:302] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86774cbe6bfa4a2ab69a2d752574e46c; no voters:
I20260925 10:25:50.910229 18638 leader_election.cc:288] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:50.910382 18643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:50.911157 18643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [term 1 LEADER]: Becoming Leader. State: Replica: 86774cbe6bfa4a2ab69a2d752574e46c, State: Running, Role: LEADER
I20260925 10:25:50.911690 18643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } }
I20260925 10:25:50.912456 18638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:50.914779 18644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } } }
I20260925 10:25:50.915249 18644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:50.917632 18645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86774cbe6bfa4a2ab69a2d752574e46c. Latest consensus state: current_term: 1 leader_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86774cbe6bfa4a2ab69a2d752574e46c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35929 } } }
I20260925 10:25:50.919880 18645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:50.920356 18652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:50.925266 18652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:50.934253 18652 catalog_manager.cc:1347] Generated new cluster ID: 6bea0cca5d384fbfac6e26945bd423fc
I20260925 10:25:50.934383 18652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:50.946058 18652 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:50.947880 18652 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:50.961138 18652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86774cbe6bfa4a2ab69a2d752574e46c: Generated new TSK 0
I20260925 10:25:50.962086 18652 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:25:51.011709 18668 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:25:51.011849 18668 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:25:51.013120 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:50.984142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.12.248.126:35929)
0925 10:25:50.984349 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:50.984356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:50.985048 (+ 692us) server_negotiation.cc:407] Connection header received
0925 10:25:50.985301 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:50.985308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:50.985419 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:50.985688 (+ 269us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:25:50.986411 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.987252 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.989409 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.989951 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.990052 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.990671 (+ 619us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:25:51.012263 (+ 21592us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:25:51.012739 (+ 476us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 10:25:51.013275 18641 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35544: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 10:25:51.013490 18667 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:50.983603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35929 (local address 127.0.0.1:35544)
0925 10:25:50.984902 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:50.984923 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:50.985096 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:50.985867 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:50.985874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:50.985888 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:25:50.986199 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:50.986213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.987400 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.987406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:50.989248 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:50.989264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:50.990141 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:50.990149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:50.990359 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:50.990408 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:25:51.012553 (+ 22145us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:25:51.012936 (+ 383us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.248.126:35929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43121/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":841,"threads_started":1}
I20260925 10:25:51.014890 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18571
I20260925 10:25:51.016168 18633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:51.114878 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18571
2026-09-25T10:25:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (764 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:25:51.170130 13281 mini_oidc.cc:152] Starting JWKS server
I20260925 10:25:51.171998 13281 webserver.cc:460] Webserver started at https://127.0.0.1:44217/ using document root <none> and password file <none>
I20260925 10:25:51.172253 13281 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:25:51.172878 13281 webserver.cc:460] Webserver started at http://127.0.0.1:44751/ using document root <none> and password file <none>
2026-09-25T10:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:51Z Disabled control of system clock
I20260925 10:25:51.199412 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:33299
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:33299
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:25:51.525540 18675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:51.526029 18675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:51.526204 18675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:51.533746 18675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:51.533917 18675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:51.533986 18675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:51.534070 18675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:51.534169 18675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:25:51.534260 18675 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:25:51.534345 18675 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44217/jwks.json
I20260925 10:25:51.543767 18675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:33299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:51.545195 18675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:51.547237 18675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:51.559500 18681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:51.559559 18683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:51.559587 18680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:51.560719 18675 server_base.cc:1034] running on GCE node
I20260925 10:25:51.561703 18675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:51.563174 18675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:51.564421 18675 hybrid_clock.cc:648] HybridClock initialized: now 1790331951564379 us; error 40 us; skew 500 ppm
I20260925 10:25:51.564867 18675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:51.567299 18675 webserver.cc:460] Webserver started at http://127.12.248.126:35729/ using document root <none> and password file <none>
I20260925 10:25:51.568029 18675 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:51.568161 18675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:51.568493 18675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:51.571484 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "4b97b8885588445ba9871ba9144e13c3"
format_stamp: "Formatted at 2026-09-25 10:25:51 on dist-test-slave-f9rl"
server_key: "21708c97b1f04e65f92ccd2bf1d137b8"
server_key_iv: "90f445ec81371baf5ed63dde031ec5dd"
server_key_version: "encryptionkey@0"
I20260925 10:25:51.572304 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "4b97b8885588445ba9871ba9144e13c3"
format_stamp: "Formatted at 2026-09-25 10:25:51 on dist-test-slave-f9rl"
server_key: "21708c97b1f04e65f92ccd2bf1d137b8"
server_key_iv: "90f445ec81371baf5ed63dde031ec5dd"
server_key_version: "encryptionkey@0"
I20260925 10:25:51.577751 18675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 10:25:51.581169 18689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:51.582659 18675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:51.582857 18675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "4b97b8885588445ba9871ba9144e13c3"
format_stamp: "Formatted at 2026-09-25 10:25:51 on dist-test-slave-f9rl"
server_key: "21708c97b1f04e65f92ccd2bf1d137b8"
server_key_iv: "90f445ec81371baf5ed63dde031ec5dd"
server_key_version: "encryptionkey@0"
I20260925 10:25:51.583099 18675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:51.612599 18675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:51.613564 18675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:51.613903 18675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:51.633252 18675 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:33299
I20260925 10:25:51.633335 18741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:33299 every 8 connection(s)
I20260925 10:25:51.635056 18675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:51.638211 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18675
I20260925 10:25:51.638447 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:51.639238 13281 external_mini_cluster.cc:1428] Setting key 0b5aa6bd9bda644fd306e701dbfb1d92
I20260925 10:25:51.639447 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:51.652011 18742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3: Bootstrap starting.
I20260925 10:25:51.655819 18742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:51.656942 18742 log.cc:824] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:51.657084 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:51.641482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.12.248.126:33299)
0925 10:25:51.646250 (+ 4768us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:51.646260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:51.646311 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:25:51.647937 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:51.647961 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:51.648471 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:51.648829 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:51.649498 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.650827 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.652637 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.653308 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.653501 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:51.654779 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:51.654786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:51.654799 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:51.655344 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:51.655431 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:51.655452 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:51.655547 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:51.655790 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:25:51.656098 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4331,"thread_start_us":4036,"threads_started":1}
I20260925 10:25:51.659967 18742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3: No bootstrap required, opened a new log
I20260925 10:25:51.661017 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:25:51.668640 18742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } }
I20260925 10:25:51.669051 18742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:51.669179 18742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b97b8885588445ba9871ba9144e13c3, State: Initialized, Role: FOLLOWER
I20260925 10:25:51.669898 18742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } }
I20260925 10:25:51.670145 18742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:51.670293 18742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:51.670466 18742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:51.672783 18742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } }
I20260925 10:25:51.673275 18742 leader_election.cc:302] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b97b8885588445ba9871ba9144e13c3; no voters:
I20260925 10:25:51.673928 18742 leader_election.cc:288] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:51.674095 18747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:51.675177 18747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b97b8885588445ba9871ba9144e13c3, State: Running, Role: LEADER
I20260925 10:25:51.675721 18747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } }
I20260925 10:25:51.676229 18742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:51.678643 18749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b97b8885588445ba9871ba9144e13c3. Latest consensus state: current_term: 1 leader_uuid: "4b97b8885588445ba9871ba9144e13c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } } }
I20260925 10:25:51.678864 18748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b97b8885588445ba9871ba9144e13c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97b8885588445ba9871ba9144e13c3" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33299 } } }
I20260925 10:25:51.679086 18749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:51.679152 18748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:51.681571 18754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:51.686872 18754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:51.696722 18754 catalog_manager.cc:1347] Generated new cluster ID: 647a789dfbc94ffa8ad2d21fa91e9723
I20260925 10:25:51.696846 18754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:51.714293 18754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:51.716159 18754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:51.732725 18754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b97b8885588445ba9871ba9144e13c3: Generated new TSK 0
I20260925 10:25:51.733809 18754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:25:51.770754 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:51.761685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.12.248.126:33299)
0925 10:25:51.761910 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:51.761917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:51.762422 (+ 505us) server_negotiation.cc:407] Connection header received
0925 10:25:51.762651 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:51.762658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:51.762756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:51.762933 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:25:51.763463 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.764587 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.766328 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.766999 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.767140 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:51.767704 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:25:51.770086 (+ 2382us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:25:51.770425 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 10:25:51.770913 18745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 10:25:51.773105 18771 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:51.761613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:33299 (local address 127.0.0.1:59328)
0925 10:25:51.762315 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:51.762332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:51.762467 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:51.762952 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:51.762956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:51.762966 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:25:51.763304 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:51.763314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.764744 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.764748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:51.766118 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:51.766187 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:51.767248 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:51.767254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:51.767394 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:51.767430 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:25:51.770318 (+ 2888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:25:51.772584 (+ 2266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.248.126:33299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44217/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":1928,"thread_start_us":377,"threads_started":1}
I20260925 10:25:51.774425 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18675
I20260925 10:25:51.775907 18737 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 10:25:51.784844 18772 webserver.cc:533] Webserver: sslize failed:
E20260925 10:25:51.784951 18772 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:25:51.872859 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18675
2026-09-25T10:25:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:51Z Disabled control of system clock
I20260925 10:25:51.955436 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40487
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:40487
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/testchainca.cert with env {}
W20260925 10:25:52.293673 18777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:52.293972 18777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:52.294059 18777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:52.301383 18777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:52.301558 18777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:52.301616 18777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:52.301674 18777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:52.301719 18777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:25:52.301774 18777 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:25:52.301813 18777 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:25:52.312027 18777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:40487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:52.313468 18777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:52.314707 18777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:52.327121 18782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:52.327201 18785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:52.327112 18783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:52.328132 18777 server_base.cc:1034] running on GCE node
I20260925 10:25:52.328884 18777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:52.330459 18777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:52.331724 18777 hybrid_clock.cc:648] HybridClock initialized: now 1790331952331663 us; error 52 us; skew 500 ppm
I20260925 10:25:52.332104 18777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:25:52.334501 18777 webserver.cc:460] Webserver started at http://127.12.248.126:40655/ using document root <none> and password file <none>
I20260925 10:25:52.335207 18777 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:52.335348 18777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:52.335666 18777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:52.338615 18777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "a48ee50590e24397b7e1ae1b590f120d"
format_stamp: "Formatted at 2026-09-25 10:25:52 on dist-test-slave-f9rl"
server_key: "663b95b07a392058702860fd5053a698"
server_key_iv: "c94f66a0d84cd494160fbb8ff5f3e390"
server_key_version: "encryptionkey@0"
I20260925 10:25:52.339465 18777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "a48ee50590e24397b7e1ae1b590f120d"
format_stamp: "Formatted at 2026-09-25 10:25:52 on dist-test-slave-f9rl"
server_key: "663b95b07a392058702860fd5053a698"
server_key_iv: "c94f66a0d84cd494160fbb8ff5f3e390"
server_key_version: "encryptionkey@0"
I20260925 10:25:52.344493 18777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 10:25:52.347826 18791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:52.349153 18777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:25:52.349349 18777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "a48ee50590e24397b7e1ae1b590f120d"
format_stamp: "Formatted at 2026-09-25 10:25:52 on dist-test-slave-f9rl"
server_key: "663b95b07a392058702860fd5053a698"
server_key_iv: "c94f66a0d84cd494160fbb8ff5f3e390"
server_key_version: "encryptionkey@0"
I20260925 10:25:52.349633 18777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:52.372258 18777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:52.373131 18777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:52.373430 18777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:52.392568 18777 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:40487
I20260925 10:25:52.392586 18842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:40487 every 8 connection(s)
I20260925 10:25:52.394028 18777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:52.394693 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18777
I20260925 10:25:52.394935 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:52.395745 13281 external_mini_cluster.cc:1428] Setting key 4c11bf9a50130a725a024ad77a798cb2
I20260925 10:25:52.398650 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:52.409144 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:52.397789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.12.248.126:40487)
0925 10:25:52.398803 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:52.398815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:52.398857 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:25:52.400448 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:52.400469 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:52.400929 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:52.401351 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:52.401949 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.403313 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.404614 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.405274 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.405492 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:52.406763 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:52.406770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:52.406788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:25:52.407373 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:52.407456 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:25:52.407480 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:52.407622 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:52.407885 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:25:52.408189 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260925 10:25:52.409871 18843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d: Bootstrap starting.
I20260925 10:25:52.413209 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:25:52.413673 18843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:52.414659 18843 log.cc:824] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:52.417853 18843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d: No bootstrap required, opened a new log
I20260925 10:25:52.425473 18843 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } }
I20260925 10:25:52.425904 18843 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:52.426017 18843 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48ee50590e24397b7e1ae1b590f120d, State: Initialized, Role: FOLLOWER
I20260925 10:25:52.426554 18843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } }
I20260925 10:25:52.426798 18843 raft_consensus.cc:388] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:52.426944 18843 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:52.427119 18843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:52.429309 18843 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } }
I20260925 10:25:52.429981 18843 leader_election.cc:302] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48ee50590e24397b7e1ae1b590f120d; no voters:
I20260925 10:25:52.430662 18843 leader_election.cc:288] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:52.430788 18848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:52.431501 18848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [term 1 LEADER]: Becoming Leader. State: Replica: a48ee50590e24397b7e1ae1b590f120d, State: Running, Role: LEADER
I20260925 10:25:52.432013 18848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } }
I20260925 10:25:52.432881 18843 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:52.434860 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48ee50590e24397b7e1ae1b590f120d. Latest consensus state: current_term: 1 leader_uuid: "a48ee50590e24397b7e1ae1b590f120d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } } }
I20260925 10:25:52.434998 18849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a48ee50590e24397b7e1ae1b590f120d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48ee50590e24397b7e1ae1b590f120d" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40487 } } }
I20260925 10:25:52.435297 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:52.435329 18849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:52.436929 18853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:52.441234 18853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:52.451212 18853 catalog_manager.cc:1347] Generated new cluster ID: 24fa0bc48fe7431a86bc4abe2f03ea1a
I20260925 10:25:52.451407 18853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:52.464754 18853 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:52.466600 18853 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:52.484740 18853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48ee50590e24397b7e1ae1b590f120d: Generated new TSK 0
I20260925 10:25:52.485981 18853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:25:52.538064 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:52.504964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.12.248.126:40487)
0925 10:25:52.505237 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:52.505244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:52.505653 (+ 409us) server_negotiation.cc:407] Connection header received
0925 10:25:52.505965 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:52.505969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:52.506060 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:52.506214 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:25:52.506737 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.507483 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.509548 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.509984 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.510078 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:52.510682 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:25:52.537151 (+ 26469us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:25:52.537727 (+ 576us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58910: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 10:25:52.538220 18846 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58910: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 10:25:52.538301 18872 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:52.504852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40487 (local address 127.0.0.1:58910)
0925 10:25:52.505531 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:52.505551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:52.505733 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:52.506252 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:52.506258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:52.506272 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:25:52.506582 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:52.506594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.507635 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.507640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:52.509374 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:52.509388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:52.510157 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:52.510166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:52.510370 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:52.510418 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:25:52.537555 (+ 27137us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:25:52.537890 (+ 335us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.248.126:40487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":340,"threads_started":1}
I20260925 10:25:52.539610 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18777
I20260925 10:25:52.540886 18838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:52.635604 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18777
2026-09-25T10:25:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (741 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (420 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1486 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:54 dist-test-slave-f9rl krb5kdc[18888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:54 dist-test-slave-f9rl krb5kdc[18888](info): set up 2 sockets
Sep 25 10:25:54 dist-test-slave-f9rl krb5kdc[18888](info): commencing operation
krb5kdc: starting...
W20260925 10:25:55.659686 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:55 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:25:55.877805 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:42883
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:42883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:56.212559 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:56.212877 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:56.213055 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:56.220474 18901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:25:56.220599 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:56.220721 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:56.220819 18901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:25:56.220909 18901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:25:56.231110 18901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:42883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:42883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:56.232528 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:56.233932 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:56.246670 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:56.246735 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:56.246817 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:56.247531 18901 server_base.cc:1034] running on GCE node
I20260925 10:25:56.248395 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:56.249871 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:56.251075 18901 hybrid_clock.cc:648] HybridClock initialized: now 1790331956251025 us; error 40 us; skew 500 ppm
I20260925 10:25:56.251436 18901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:56 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:56.260331 18901 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:25:56.262029 18901 webserver.cc:460] Webserver started at http://127.12.248.126:35553/ using document root <none> and password file <none>
I20260925 10:25:56.262742 18901 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:56.262889 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:56.263211 18901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:56.266484 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "e987ea63a436413ead3f2e089e9adf61"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "e4b975bd07c680b14b9563eb245b564b"
server_key_iv: "fbbbacce71ea0d9a43be7486c4594492"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.267269 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "e987ea63a436413ead3f2e089e9adf61"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "e4b975bd07c680b14b9563eb245b564b"
server_key_iv: "fbbbacce71ea0d9a43be7486c4594492"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.272513 18901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:25:56.276176 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:56.277671 18901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:25:56.277873 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "e987ea63a436413ead3f2e089e9adf61"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "e4b975bd07c680b14b9563eb245b564b"
server_key_iv: "fbbbacce71ea0d9a43be7486c4594492"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.278108 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:56.299513 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:56.303786 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:56.304103 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:56.329185 18901 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:42883
I20260925 10:25:56.329218 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:42883 every 8 connection(s)
I20260925 10:25:56.331310 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:25:56.336615 18968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:25:56.339399 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18901
I20260925 10:25:56.339632 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:25:56.340546 13281 external_mini_cluster.cc:1428] Setting key ce935f972decaa9b61bf49c10e717c61
I20260925 10:25:56.354812 18968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61: Bootstrap starting.
I20260925 10:25:56.358987 18968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61: Neither blocks nor log segments found. Creating new log.
I20260925 10:25:56.360234 18968 log.cc:824] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61: Log is configured to *not* fsync() on all Append() calls
I20260925 10:25:56.363318 18968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61: No bootstrap required, opened a new log
Sep 25 10:25:56 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:56.372939 18968 raft_consensus.cc:348] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } }
I20260925 10:25:56.373485 18968 raft_consensus.cc:374] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:25:56.373606 18968 raft_consensus.cc:729] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987ea63a436413ead3f2e089e9adf61, State: Initialized, Role: FOLLOWER
I20260925 10:25:56.374253 18968 consensus_queue.cc:260] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } }
I20260925 10:25:56.374505 18968 raft_consensus.cc:388] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:25:56.374670 18968 raft_consensus.cc:482] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:25:56.374853 18968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:25:56.377244 18968 raft_consensus.cc:504] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } }
I20260925 10:25:56.377782 18968 leader_election.cc:302] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e987ea63a436413ead3f2e089e9adf61; no voters:
I20260925 10:25:56.378566 18968 leader_election.cc:288] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:25:56.378684 18973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:25:56.380296 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:56.342994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.12.248.126:42883)
0925 10:25:56.346437 (+ 3443us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:56.346456 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:56.346514 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:25:56.349388 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:56.349427 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:56.350095 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:56.350629 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:56.352309 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.354688 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.356986 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.358106 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.358313 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:56.371704 (+ 13391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:56.371713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:56.371732 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:56.371774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:56.375635 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:56.376366 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:56.376375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:56.376387 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:56.376559 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:56.376986 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:56.376991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:56.376994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:56.377641 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:56.377945 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:56.378416 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:25:56.378908 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3142,"thread_start_us":2897,"threads_started":1}
I20260925 10:25:56.381338 18968 sys_catalog.cc:564] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:25:56.382232 18973 raft_consensus.cc:686] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [term 1 LEADER]: Becoming Leader. State: Replica: e987ea63a436413ead3f2e089e9adf61, State: Running, Role: LEADER
I20260925 10:25:56.382897 18973 consensus_queue.cc:237] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } }
I20260925 10:25:56.386508 18975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e987ea63a436413ead3f2e089e9adf61. Latest consensus state: current_term: 1 leader_uuid: "e987ea63a436413ead3f2e089e9adf61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } } }
I20260925 10:25:56.386605 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e987ea63a436413ead3f2e089e9adf61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e987ea63a436413ead3f2e089e9adf61" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42883 } } }
I20260925 10:25:56.386950 18975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:56.387012 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61 [sys.catalog]: This master's current role is: LEADER
I20260925 10:25:56.390086 18978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:25:56.395428 18978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:25:56.404814 18978 catalog_manager.cc:1347] Generated new cluster ID: 62c64dcf9601419a935aa114ceae2517
I20260925 10:25:56.404948 18978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:25:56.424644 18978 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:25:56.426117 18978 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:25:56.432888 18978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e987ea63a436413ead3f2e089e9adf61: Generated new TSK 0
I20260925 10:25:56.433804 18978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:25:56.489814 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:56.826752 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:56.827018 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:56.827114 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:56.834164 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:56.834271 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:56.834386 18994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:25:56.843900 18994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:56.845168 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:56.846475 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:56.859449 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:56.859493 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:56.859529 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:56.860353 18994 server_base.cc:1034] running on GCE node
I20260925 10:25:56.860980 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:56.862044 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:56.863291 18994 hybrid_clock.cc:648] HybridClock initialized: now 1790331956863218 us; error 58 us; skew 500 ppm
I20260925 10:25:56.863662 18994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:56 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:56.867038 18994 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:25:56.868602 18994 webserver.cc:460] Webserver started at http://127.12.248.65:35441/ using document root <none> and password file <none>
I20260925 10:25:56.869333 18994 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:56.869488 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:56.869827 18994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:56.872838 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "f4013682c44e419f819d1bb294c0edc8"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "0283cc7edb2dd42dfd4f6d23fb127be4"
server_key_iv: "4a99b0ad613923e6b409b6ed77f4bb68"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.873721 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "f4013682c44e419f819d1bb294c0edc8"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "0283cc7edb2dd42dfd4f6d23fb127be4"
server_key_iv: "4a99b0ad613923e6b409b6ed77f4bb68"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.879225 18994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:25:56.883179 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:56.884483 18994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:56.884652 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "f4013682c44e419f819d1bb294c0edc8"
format_stamp: "Formatted at 2026-09-25 10:25:56 on dist-test-slave-f9rl"
server_key: "0283cc7edb2dd42dfd4f6d23fb127be4"
server_key_iv: "4a99b0ad613923e6b409b6ed77f4bb68"
server_key_version: "encryptionkey@0"
I20260925 10:25:56.884871 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:56.913869 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:56.918427 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:56.918725 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:56.919703 18994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:56.921052 18994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:56.921154 18994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:56.921303 18994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:56.921375 18994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:56.955360 18994 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:41327
I20260925 10:25:56.955449 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:41327 every 8 connection(s)
I20260925 10:25:56.956686 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:25:56.963601 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 18994
I20260925 10:25:56.963790 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:25:56.964567 13281 external_mini_cluster.cc:1428] Setting key 28a9e654f107fe07d7654709d13851ce
Sep 25 10:25:56 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331956, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:56.977677 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:56.959338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:50247 (local address 127.12.248.126:42883)
0925 10:25:56.959803 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:56.959810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:56.960748 (+ 938us) server_negotiation.cc:407] Connection header received
0925 10:25:56.962574 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:56.962578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:56.962645 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:56.962740 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:56.964268 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.965328 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.967081 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.967479 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.967584 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:56.971819 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:56.971825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:56.971828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:56.971861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:56.974498 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:56.975216 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:56.975226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:56.975228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:56.975371 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:56.975809 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:56.975818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:56.975820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:56.976071 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:56.976226 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:56.977059 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 10:25:56.977293 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260925 10:25:56.978757 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:56.959887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42883 (local address 127.12.248.65:50247)
0925 10:25:56.960583 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:56.960624 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:56.962333 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:56.962952 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:56.962959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:56.963429 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:56.964093 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:56.964112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.965506 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.965518 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:56.966938 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:56.966953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:56.967721 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:56.967732 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:56.968081 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:56.968866 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:56.968900 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:56.971577 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:56.974636 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:56.974646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:56.974664 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:56.975085 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:56.975484 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:56.975489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:56.975493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:56.975680 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:56.976199 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:56.976207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:56.976615 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 10:25:56.976949 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 10:25:56.977511 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":279,"threads_started":1}
I20260925 10:25:56.980389 19122 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42883
I20260925 10:25:56.980724 19122 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:56.981668 19122 heartbeater.cc:511] Master 127.12.248.126:42883 requested a full tablet report, sending...
I20260925 10:25:56.984434 18933 ts_manager.cc:194] Registered new tserver with Master: f4013682c44e419f819d1bb294c0edc8 (127.12.248.65:41327)
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:25:56.986500 18933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:50247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:25:57.013823 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:57.344043 19128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:57.344393 19128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:57.344571 19128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:57.352073 19128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:57.352236 19128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:57.352427 19128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:25:57.362931 19128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:57.364298 19128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:57.365808 19128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:57.379268 19133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:57.379245 19134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:57.379206 19136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:57.380084 19128 server_base.cc:1034] running on GCE node
I20260925 10:25:57.380689 19128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:57.381829 19128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:57.383054 19128 hybrid_clock.cc:648] HybridClock initialized: now 1790331957383006 us; error 40 us; skew 500 ppm
I20260925 10:25:57.383383 19128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:57 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:57.386809 19128 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:25:57.388439 19128 webserver.cc:460] Webserver started at http://127.12.248.66:36789/ using document root <none> and password file <none>
I20260925 10:25:57.389205 19128 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:57.389384 19128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:57.389724 19128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:57.393154 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "b7179478ee8f4e6d81c3e7ad45225a1b"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "c2fd888d76a2cd5de23d2afcf4a8b2a4"
server_key_iv: "bd05720f8544adb20289c2d8dabd1666"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.394091 19128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "b7179478ee8f4e6d81c3e7ad45225a1b"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "c2fd888d76a2cd5de23d2afcf4a8b2a4"
server_key_iv: "bd05720f8544adb20289c2d8dabd1666"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.399346 19128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:25:57.402798 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.404102 19128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:25:57.404289 19128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "b7179478ee8f4e6d81c3e7ad45225a1b"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "c2fd888d76a2cd5de23d2afcf4a8b2a4"
server_key_iv: "bd05720f8544adb20289c2d8dabd1666"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.404491 19128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:57.432981 19128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:57.437529 19128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:57.437836 19128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:57.438851 19128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:57.440207 19128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:57.440318 19128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.440469 19128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:57.440544 19128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.474301 19128 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:46133
I20260925 10:25:57.474403 19255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:46133 every 8 connection(s)
I20260925 10:25:57.475683 19128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:25:57.484299 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19128
I20260925 10:25:57.484488 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:25:57.485033 13281 external_mini_cluster.cc:1428] Setting key e8d7a2a75c88e777c81700d6de82988e
Sep 25 10:25:57 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:57.499480 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:57.478621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:51695 (local address 127.12.248.126:42883)
0925 10:25:57.479092 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:57.479099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:57.480269 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 10:25:57.482364 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:57.482369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:57.482443 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:57.482548 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:57.484151 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.485298 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.487517 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.488207 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.488351 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:57.492519 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:57.492524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:57.492527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:57.492559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:57.496136 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:57.496889 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:57.496897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:57.496903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:57.497076 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:57.497541 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:57.497549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:57.497554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:57.497932 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:57.498082 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:57.498730 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 10:25:57.498977 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260925 10:25:57.500116 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:57.479278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42883 (local address 127.12.248.66:51695)
0925 10:25:57.480093 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:57.480141 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:57.482057 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:57.482792 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:57.482801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:57.483391 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:57.483961 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:57.483980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.485486 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.485495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:57.487381 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:57.487392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.488412 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.488417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:57.488666 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:57.489384 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:57.489407 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:57.492285 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:57.496282 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:57.496289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:57.496303 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:57.496726 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:57.497185 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:57.497188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:57.497190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:57.497371 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:57.498051 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:57.498056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:57.498361 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 10:25:57.498580 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 10:25:57.498991 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":346,"threads_started":1}
I20260925 10:25:57.501745 19256 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42883
I20260925 10:25:57.502056 19256 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:57.502830 19256 heartbeater.cc:511] Master 127.12.248.126:42883 requested a full tablet report, sending...
I20260925 10:25:57.504397 18933 ts_manager.cc:194] Registered new tserver with Master: b7179478ee8f4e6d81c3e7ad45225a1b (127.12.248.66:46133)
I20260925 10:25:57.505201 18933 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:51695
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:25:57.534219 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:25:57.858310 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:25:57.858700 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:25:57.858879 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:25:57.866355 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:25:57.866514 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:25:57.866696 19262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:25:57.876768 19262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:25:57.878283 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:25:57.879698 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:25:57.893735 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:57.893949 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:25:57.893888 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:25:57.894891 19262 server_base.cc:1034] running on GCE node
I20260925 10:25:57.895638 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:25:57.896703 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:25:57.897995 19262 hybrid_clock.cc:648] HybridClock initialized: now 1790331957897950 us; error 50 us; skew 500 ppm
I20260925 10:25:57.898408 19262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:25:57 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:57.901749 19262 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:25:57.903450 19262 webserver.cc:460] Webserver started at http://127.12.248.67:34723/ using document root <none> and password file <none>
I20260925 10:25:57.904222 19262 fs_manager.cc:359] Metadata directory not provided
I20260925 10:25:57.904381 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:25:57.904732 19262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:25:57.908236 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "a36e348fe1e04dd6adab42fb631e472c"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "08186e4578ca4a185c6680fa9b88bffc"
server_key_iv: "7b70fcce4fedf520702da987a088633f"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.909205 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "a36e348fe1e04dd6adab42fb631e472c"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "08186e4578ca4a185c6680fa9b88bffc"
server_key_iv: "7b70fcce4fedf520702da987a088633f"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.914767 19262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 10:25:57.918485 19277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.919909 19262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:25:57.920105 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "a36e348fe1e04dd6adab42fb631e472c"
format_stamp: "Formatted at 2026-09-25 10:25:57 on dist-test-slave-f9rl"
server_key: "08186e4578ca4a185c6680fa9b88bffc"
server_key_iv: "7b70fcce4fedf520702da987a088633f"
server_key_version: "encryptionkey@0"
I20260925 10:25:57.920336 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:25:57.934921 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:25:57.939128 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:25:57.939440 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:25:57.940418 19262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:25:57.941880 19262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:25:57.941991 19262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.942144 19262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:25:57.942221 19262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:25:57.975705 19262 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:45365
I20260925 10:25:57.975793 19389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:45365 every 8 connection(s)
I20260925 10:25:57.977094 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:25:57.984154 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19262
I20260925 10:25:57.984337 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:25:57.984778 13281 external_mini_cluster.cc:1428] Setting key 2232446f52e06032764caad0b1a295d6
I20260925 10:25:57.989710 19122 heartbeater.cc:503] Master 127.12.248.126:42883 was elected leader, sending a full tablet report...
Sep 25 10:25:57 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:25:57.997792 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:57.979920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:50289 (local address 127.12.248.126:42883)
0925 10:25:57.980106 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:57.980111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:57.981294 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 10:25:57.983318 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:57.983322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:57.983407 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:57.983495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:57.985089 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.986098 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.987649 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.988053 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.988193 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:57.992414 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:57.992419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:57.992422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:57.992454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:57.995129 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:57.995795 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:57.995799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:57.995801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:57.995917 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:57.996319 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:57.996322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:57.996324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:57.996572 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:57.996695 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:57.997316 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 10:25:57.997542 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:25:57.998839 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:57.980331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42883 (local address 127.12.248.67:50289)
0925 10:25:57.981113 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 10:25:57.981159 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:25:57.983071 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:25:57.983690 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:25:57.983697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:25:57.984203 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:25:57.984894 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:57.984919 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.986251 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.986257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:57.987495 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:25:57.987507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:25:57.988284 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:25:57.988291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:25:57.988545 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:57.989243 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:25:57.989271 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:25:57.992171 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:25:57.995267 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:57.995274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:57.995287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:57.995694 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:57.996014 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:25:57.996017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:25:57.996019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:25:57.996197 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:25:57.996674 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:25:57.996680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:25:57.997003 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 10:25:57.997231 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 10:25:57.997711 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":274,"threads_started":1}
I20260925 10:25:58.000334 19390 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42883
I20260925 10:25:58.000672 19390 heartbeater.cc:464] Registering TS with master...
I20260925 10:25:58.001510 19390 heartbeater.cc:511] Master 127.12.248.126:42883 requested a full tablet report, sending...
I20260925 10:25:58.002898 18932 ts_manager.cc:194] Registered new tserver with Master: a36e348fe1e04dd6adab42fb631e472c (127.12.248.67:45365)
I20260925 10:25:58.003754 18932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:50289
I20260925 10:25:58.012859 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/test-admin.keytab.
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.039582 13281 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 10:25:58.045205 19397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:58.042396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.12.248.65:41327)
0925 10:25:58.043092 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.043102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.043136 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:25:58.043318 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.043325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.043550 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.043739 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.044748 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58644: BlockingRecv error: recv got EOF from 127.0.0.1:58644 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":309,"threads_started":1}
W20260925 10:25:58.045300 13281 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.046922 13281 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.048377 13281 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:58.063542 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.050144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.12.248.65:41327)
0925 10:25:58.050310 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.050314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.050567 (+ 253us) server_negotiation.cc:407] Connection header received
0925 10:25:58.050703 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.050706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.050775 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.050861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.051862 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.052833 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.054128 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.054675 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.054779 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.058397 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.058409 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.058425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.058465 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.061022 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.061735 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.061743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.061745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.061877 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.062212 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.062215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.062217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.062568 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.062817 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.063046 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.063266 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:25:58.069320 19397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:58.067768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.12.248.65:41327)
0925 10:25:58.067966 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.067973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.068124 (+ 151us) server_negotiation.cc:407] Connection header received
0925 10:25:58.068250 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.068253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.068307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.068381 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.069064 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58664: BlockingRecv error: recv got EOF from 127.0.0.1:58664 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 10:25:58.069484 13281 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.071018 13281 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.072294 13281 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:58.086669 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.073721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.12.248.65:41327)
0925 10:25:58.073947 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.073953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.074143 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:25:58.074340 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.074354 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.074410 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.074515 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.075487 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.076200 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.077553 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.077984 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.078116 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.081760 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.081765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.081769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.081814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.084453 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.085052 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.085057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.085059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.085181 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.085571 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.085575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.085577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.085879 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.086081 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.086283 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.086439 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 10:25:58.091600 19397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:25:58.089637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.12.248.65:41327)
0925 10:25:58.089896 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.089905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.090189 (+ 284us) server_negotiation.cc:407] Connection header received
0925 10:25:58.090406 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.090411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.090489 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.090592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.091351 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58684: BlockingRecv error: recv got EOF from 127.0.0.1:58684 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260925 10:25:58.091854 13281 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331906218802-13281-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.093612 13281 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:25:58.095242 13281 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:25:58.110437 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.096633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.12.248.65:41327)
0925 10:25:58.096844 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.096850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.097032 (+ 182us) server_negotiation.cc:407] Connection header received
0925 10:25:58.097157 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.097161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.097216 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.097297 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.098311 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.099044 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.100312 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.100738 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.100835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.104634 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.104639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.104643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.104677 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.107213 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.108013 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.108018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.108020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.108293 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.108662 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.108666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.108668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.109038 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.109323 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.109850 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.110068 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:25:58.112329 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18994
I20260925 10:25:58.125808 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.113205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.12.248.65:41327)
0925 10:25:58.113412 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.113421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.113765 (+ 344us) server_negotiation.cc:407] Connection header received
0925 10:25:58.113985 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.113989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.114048 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.114156 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.115454 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.116149 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.117340 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.117842 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.117982 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.120150 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.120155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.120159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.120189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.122696 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.123377 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.123385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.123393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.123593 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.124044 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.124049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.124052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.124478 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.124761 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.125215 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.125461 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:25:58.126374 19117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:58.210156 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18994
I20260925 10:25:58.226676 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19128
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:25:58.242055 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.227668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.12.248.66:46133)
0925 10:25:58.228223 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.228230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.228256 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:25:58.228315 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.228321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.228530 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.228663 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.229673 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.230733 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.231904 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.232458 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.232555 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.236352 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.236363 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.236377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.236409 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.239464 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.240104 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.240110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.240112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.240219 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.240575 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.240578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.240580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.240889 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.241109 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.241404 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.241650 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260925 10:25:58.243571 19251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:58.326812 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19128
I20260925 10:25:58.347538 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19262
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:25:58.362769 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:25:58.348573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.12.248.67:45365)
0925 10:25:58.349221 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0925 10:25:58.349228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:25:58.349259 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:25:58.349316 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:25:58.349322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:25:58.349600 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:25:58.349731 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:25:58.350756 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.351751 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.352918 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:25:58.353506 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:25:58.353627 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:25:58.357493 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:25:58.357513 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:25:58.357528 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:25:58.357562 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:25:58.360243 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.360868 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.360875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.360877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.360990 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:25:58.361320 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:25:58.361323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:25:58.361325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:25:58.361717 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:25:58.361943 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:25:58.362190 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 10:25:58.362438 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":359,"threads_started":1}
I20260925 10:25:58.364073 19385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:58.447738 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19262
I20260925 10:25:58.463379 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 18901
I20260925 10:25:58.464706 18963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:25:58.568694 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 18901
2026-09-25T10:25:58Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4032 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[19417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[19417](info): set up 2 sockets
Sep 25 10:25:58 dist-test-slave-f9rl krb5kdc[19417](info): commencing operation
krb5kdc: starting...
W20260925 10:25:59.667733 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:25:59 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:25:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab.
Entry for principal oryx/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab.
I20260925 10:25:59.887578 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38783
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab
--principal=oryx/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:00.222114 19430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:00.222433 19430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:00.222605 19430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:00.229977 19430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:00.230089 19430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:00.230204 19430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:00.230285 19430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:00.230351 19430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:00.240453 19430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab
--principal=oryx/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:00.241868 19430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:00.243278 19430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:00.256004 19435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:00.256014 19438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:00.256042 19436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:00.257085 19430 server_base.cc:1034] running on GCE node
I20260925 10:26:00.257949 19430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:00.259514 19430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:00.260741 19430 hybrid_clock.cc:648] HybridClock initialized: now 1790331960260670 us; error 41 us; skew 500 ppm
I20260925 10:26:00.261102 19430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:00 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331960, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:00.264495 19430 init.cc:377] Logged in from keytab as oryx/127.12.248.126@KRBTEST.COM (short username oryx)
I20260925 10:26:00.266119 19430 webserver.cc:460] Webserver started at http://127.12.248.126:38535/ using document root <none> and password file <none>
I20260925 10:26:00.266839 19430 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:00.266985 19430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:00.267297 19430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:00.270469 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "110d185250964711b03a7dc16bceaf87"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "7a77162d2a9c450533d1e8579894005a"
server_key_iv: "e3f9349339ad5447cff457661baf2478"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.271262 19430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "110d185250964711b03a7dc16bceaf87"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "7a77162d2a9c450533d1e8579894005a"
server_key_iv: "e3f9349339ad5447cff457661baf2478"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.276592 19430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:26:00.280220 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:00.281733 19430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:26:00.281915 19430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "110d185250964711b03a7dc16bceaf87"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "7a77162d2a9c450533d1e8579894005a"
server_key_iv: "e3f9349339ad5447cff457661baf2478"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.282091 19430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:00.296114 19430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:00.300542 19430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:00.300853 19430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:00.320832 19430 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:38783
I20260925 10:26:00.320858 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:38783 every 8 connection(s)
I20260925 10:26:00.322417 19430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:00.326498 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:00.328691 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19430
I20260925 10:26:00.328943 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:00.329840 13281 external_mini_cluster.cc:1428] Setting key 505d3c0700b66f2f19fbc27db2be2a70
I20260925 10:26:00.339751 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Bootstrap starting.
I20260925 10:26:00.343823 19497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Neither blocks nor log segments found. Creating new log.
Sep 25 10:26:00 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:00.345096 19497 log.cc:824] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:00.348210 19497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: No bootstrap required, opened a new log
I20260925 10:26:00.353608 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:00.332079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.12.248.126:38783)
0925 10:26:00.332952 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:00.332965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:00.333007 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:26:00.334711 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:00.334735 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:00.335126 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:00.335455 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:00.337334 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.338730 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.340454 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.341109 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.341279 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:00.346101 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:00.346110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:00.346126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:00.346175 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:00.349318 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:00.350072 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:00.350079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:00.350089 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:00.350260 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:00.350666 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:00.350677 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:00.350685 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:00.351232 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:00.351577 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:00.351991 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 10:26:00.352443 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":411,"threads_started":1}
I20260925 10:26:00.356923 19497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } }
I20260925 10:26:00.357306 19497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:00.357416 19497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110d185250964711b03a7dc16bceaf87, State: Initialized, Role: FOLLOWER
I20260925 10:26:00.358134 19497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } }
I20260925 10:26:00.358378 19497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:00.358538 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:00.358714 19497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:00.361210 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } }
I20260925 10:26:00.361756 19497 leader_election.cc:302] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110d185250964711b03a7dc16bceaf87; no voters:
I20260925 10:26:00.362418 19497 leader_election.cc:288] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:00.362561 19502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:00.363710 19502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 LEADER]: Becoming Leader. State: Replica: 110d185250964711b03a7dc16bceaf87, State: Running, Role: LEADER
I20260925 10:26:00.364214 19502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } }
I20260925 10:26:00.364768 19497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:00.367163 19504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110d185250964711b03a7dc16bceaf87. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } }
I20260925 10:26:00.367591 19504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:00.367163 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } }
I20260925 10:26:00.368010 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:00.370496 19508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:00.375131 19508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:00.384006 19508 catalog_manager.cc:1347] Generated new cluster ID: 4d36fe76d3444ea7a64880ca0becff65
I20260925 10:26:00.384140 19508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:00.410434 19508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:00.412556 19508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:00.424336 19508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Generated new TSK 0
I20260925 10:26:00.425212 19508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
I20260925 10:26:00.489706 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:00.826529 19523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:00.826884 19523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:00.827054 19523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:00.834537 19523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:00.834698 19523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:00.834890 19523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:00.844954 19523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:00.846426 19523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:00.847846 19523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:00.861930 19528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:00.861901 19531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:00.862006 19529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:00.868130 19523 server_base.cc:1034] running on GCE node
I20260925 10:26:00.869042 19523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:00.870296 19523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:00.871546 19523 hybrid_clock.cc:648] HybridClock initialized: now 1790331960871504 us; error 45 us; skew 500 ppm
I20260925 10:26:00.871919 19523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:00 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331960, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:00.875460 19523 init.cc:377] Logged in from keytab as oryx/127.12.248.65@KRBTEST.COM (short username oryx)
I20260925 10:26:00.877072 19523 webserver.cc:460] Webserver started at http://127.12.248.65:33927/ using document root <none> and password file <none>
I20260925 10:26:00.877871 19523 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:00.877975 19523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:00.878228 19523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:00.881253 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "69421f70a24d423387c1b7f7d8db5e42"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "acfc9078d9fa4f04d2fe285d0434bdc3"
server_key_iv: "3d26ba06d3fdee20fbfe527e74b50db0"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.882081 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "69421f70a24d423387c1b7f7d8db5e42"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "acfc9078d9fa4f04d2fe285d0434bdc3"
server_key_iv: "3d26ba06d3fdee20fbfe527e74b50db0"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.887859 19523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:26:00.892078 19538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:00.893846 19523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:26:00.894039 19523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "69421f70a24d423387c1b7f7d8db5e42"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "acfc9078d9fa4f04d2fe285d0434bdc3"
server_key_iv: "3d26ba06d3fdee20fbfe527e74b50db0"
server_key_version: "encryptionkey@0"
I20260925 10:26:00.894284 19523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:00.916276 19523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:00.920969 19523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:00.921293 19523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:00.922377 19523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:00.923966 19523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:00.924060 19523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:00.924228 19523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:00.924290 19523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:00.958781 19523 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:39655
I20260925 10:26:00.958832 19650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:39655 every 8 connection(s)
I20260925 10:26:00.960163 19523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:00.964656 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19523
I20260925 10:26:00.964881 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:00.965812 13281 external_mini_cluster.cc:1428] Setting key 86d6ba52f3d0652ef8d402772e1e97e9
Sep 25 10:26:00 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331960, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:00.985723 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:00.962868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:60079 (local address 127.12.248.126:38783)
0925 10:26:00.963321 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:00.963326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:00.964443 (+ 1117us) server_negotiation.cc:407] Connection header received
0925 10:26:00.967017 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:00.967022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:00.967116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:00.967235 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:00.969197 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.970143 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.972800 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.973275 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.973382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:00.978136 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:00.978146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:00.978154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:00.978200 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:00.981980 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:00.982758 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:00.982767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:00.982773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:00.982921 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:00.983479 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:00.983486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:00.983492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:00.983895 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:00.984065 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:00.984877 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 10:26:00.985145 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260925 10:26:00.986837 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:00.963453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:60079)
0925 10:26:00.964230 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:00.964294 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:00.966714 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:00.967491 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:00.967500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:00.968140 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:00.968975 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:00.969003 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.970320 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.970328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:00.972627 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:00.972645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:00.973724 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:00.973733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:00.974072 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:00.975003 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:00.975046 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:00.977881 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:00.982150 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:00.982157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:00.982172 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:00.982619 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:00.983034 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:00.983037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:00.983039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:00.983368 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:00.984061 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:00.984068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:00.984431 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 10:26:00.984726 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 10:26:00.985262 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":291,"threads_started":1}
I20260925 10:26:00.988838 19651 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:00.989241 19651 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:00.990144 19651 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:26:00.992552 19462 ts_manager.cc:194] Registered new tserver with Master: 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.66@KRBTEST.COM" created.
I20260925 10:26:00.994230 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:60079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
I20260925 10:26:01.022457 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:01.362578 19657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:01.362869 19657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:01.362964 19657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:01.370333 19657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:01.370443 19657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:01.370620 19657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:01.380250 19657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:01.381541 19657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:01.382825 19657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:01.396215 19663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:01.396342 19662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:01.396308 19665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:01.397835 19657 server_base.cc:1034] running on GCE node
I20260925 10:26:01.398536 19657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:01.399526 19657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:01.400754 19657 hybrid_clock.cc:648] HybridClock initialized: now 1790331961400705 us; error 40 us; skew 500 ppm
I20260925 10:26:01.401131 19657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:01 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331961, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:01.412789 19657 init.cc:377] Logged in from keytab as oryx/127.12.248.66@KRBTEST.COM (short username oryx)
I20260925 10:26:01.414423 19657 webserver.cc:460] Webserver started at http://127.12.248.66:36453/ using document root <none> and password file <none>
I20260925 10:26:01.415096 19657 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:01.415198 19657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:01.415486 19657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:01.418430 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "7178757e204696c2de6cf41f23d5d0a2"
server_key_iv: "5740901d3c2f5ddebaedd8c8b02f550e"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.419133 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "7178757e204696c2de6cf41f23d5d0a2"
server_key_iv: "5740901d3c2f5ddebaedd8c8b02f550e"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.424189 19657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:26:01.427815 19672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.429183 19657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 10:26:01.429388 19657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "7178757e204696c2de6cf41f23d5d0a2"
server_key_iv: "5740901d3c2f5ddebaedd8c8b02f550e"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.429631 19657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:01.451851 19657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:01.456516 19657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:01.456754 19657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:01.457746 19657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:01.459108 19657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:01.459184 19657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.459276 19657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:01.459319 19657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.493230 19657 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:44403
I20260925 10:26:01.493279 19784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:44403 every 8 connection(s)
I20260925 10:26:01.494666 19657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:01.504218 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19657
I20260925 10:26:01.504397 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:01.504894 13281 external_mini_cluster.cc:1428] Setting key 5b525f540a6cbce8f446de3509fffa88
Sep 25 10:26:01 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331961, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:01.515528 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:01.497306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:39459 (local address 127.12.248.126:38783)
0925 10:26:01.497869 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:01.497873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:01.499048 (+ 1175us) server_negotiation.cc:407] Connection header received
0925 10:26:01.500905 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:01.500911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:01.501006 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:01.501127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:01.502699 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:01.503462 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:01.505223 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:01.505742 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:01.505879 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:01.509349 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:01.509354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:01.509357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:01.509391 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:01.512075 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:01.512989 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:01.512993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:01.512995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:01.513121 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:01.513680 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:01.513683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:01.513685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:01.513976 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:01.514091 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:01.515065 (+ 974us) server_negotiation.cc:299] Negotiation successful
0925 10:26:01.515251 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
I20260925 10:26:01.517151 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:01.498091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:39459)
0925 10:26:01.498889 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:01.498927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:01.500620 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:01.501371 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:01.501380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:01.501952 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:01.502517 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:01.502534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:01.503631 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:01.503635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:01.505084 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:01.505097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:01.505986 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:01.505993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:01.506221 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:01.506930 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:01.506957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:01.509097 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:01.512226 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:01.512235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:01.512254 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:01.512856 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:01.513236 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:01.513242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:01.513247 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:01.513528 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:01.514111 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:01.514122 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:01.514610 (+ 488us) client_negotiation.cc:769] Sending connection context
0925 10:26:01.514964 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 10:26:01.515590 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":278,"threads_started":1}
I20260925 10:26:01.518683 19785 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:01.519018 19785 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:01.519764 19785 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
I20260925 10:26:01.521384 19462 ts_manager.cc:194] Registered new tserver with Master: 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:01.522490 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:39459
WARNING: no policy specified for oryx/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
I20260925 10:26:01.557858 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:01.885139 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:01.885522 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:01.885694 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:01.893054 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:01.893227 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:01.893415 19791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:01.903630 19791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:01.904991 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:01.906443 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:01.919823 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:01.919866 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:01.919890 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:01.920711 19791 server_base.cc:1034] running on GCE node
I20260925 10:26:01.921344 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:01.922372 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:01.923583 19791 hybrid_clock.cc:648] HybridClock initialized: now 1790331961923546 us; error 23 us; skew 500 ppm
I20260925 10:26:01.923861 19791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:01 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331961, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:01.927129 19791 init.cc:377] Logged in from keytab as oryx/127.12.248.67@KRBTEST.COM (short username oryx)
I20260925 10:26:01.928675 19791 webserver.cc:460] Webserver started at http://127.12.248.67:41371/ using document root <none> and password file <none>
I20260925 10:26:01.929430 19791 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:01.929639 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:01.929962 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:01.933123 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "91ee94d52ad541fcbaad403a1d627840"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "1596e33475293a62fb5c5e137a3a7aec"
server_key_iv: "b63bb7c5d5da09ae8e8625e6fd3fa08c"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.934087 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "91ee94d52ad541fcbaad403a1d627840"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "1596e33475293a62fb5c5e137a3a7aec"
server_key_iv: "b63bb7c5d5da09ae8e8625e6fd3fa08c"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.939405 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:26:01.942940 19806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.944326 19791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:26:01.944504 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "91ee94d52ad541fcbaad403a1d627840"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "1596e33475293a62fb5c5e137a3a7aec"
server_key_iv: "b63bb7c5d5da09ae8e8625e6fd3fa08c"
server_key_version: "encryptionkey@0"
I20260925 10:26:01.944707 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:01.958931 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:01.963210 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:01.963505 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:01.964432 19791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:01.966075 19791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:01.966187 19791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.966311 19791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:01.966395 19791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:01.998193 19651 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
I20260925 10:26:02.001740 19791 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:41331
I20260925 10:26:02.001781 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:41331 every 8 connection(s)
I20260925 10:26:02.003072 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:02.008543 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19791
I20260925 10:26:02.008741 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:02.009333 13281 external_mini_cluster.cc:1428] Setting key 3fbcc91e5f031048d1767439501050c6
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331961, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:02.023381 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.005647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:36987 (local address 127.12.248.126:38783)
0925 10:26:02.005841 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.005846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.007083 (+ 1237us) server_negotiation.cc:407] Connection header received
0925 10:26:02.008842 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.008848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.008941 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.009052 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.010819 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.011753 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.013603 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.014035 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.014137 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.017587 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.017592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.017595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.017627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.020375 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.021075 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.021078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.021080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.021196 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.021659 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.021662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.021664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.021939 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.022062 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.022771 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.023042 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:26:02.024510 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.006202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:36987)
0925 10:26:02.006918 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.006966 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.008525 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.009251 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.009259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.009886 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.010668 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.010689 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.011924 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.011928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.013373 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.013386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.014233 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.014238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.014455 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.015162 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.015185 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.017335 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.020518 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.020524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.020538 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.020961 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.021303 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.021306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.021308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.021545 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.022053 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.022059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.022415 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.022644 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.023044 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":288,"threads_started":1}
I20260925 10:26:02.026437 19919 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:02.026727 19919 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:02.027554 19919 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
I20260925 10:26:02.028977 19462 ts_manager.cc:194] Registered new tserver with Master: 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:02.029876 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:36987
I20260925 10:26:02.038177 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.248.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.125.keytab.
Entry for principal oryx/127.12.248.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.125.keytab.
I20260925 10:26:02.088542 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:41103
--webserver_interface=127.12.248.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.125.keytab
--principal=oryx/127.12.248.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38783,127.12.248.125:41103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:02.418654 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:02.418999 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:02.419178 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:02.426680 19924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:02.426831 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:02.426942 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:02.427054 19924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:02.427135 19924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:02.437253 19924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38783,127.12.248.125:41103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.125.keytab
--principal=oryx/127.12.248.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:41103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.248.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
W20260925 10:26:02.437695 19924 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.248.126:38783,127.12.248.125:41103'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:26:02.438781 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:02.440099 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:02.452072 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:02.452102 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:02.452200 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:02.453081 19924 server_base.cc:1034] running on GCE node
I20260925 10:26:02.454075 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:02.455525 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:02.456779 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790331962456698 us; error 50 us; skew 500 ppm
I20260925 10:26:02.457111 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331962, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:02.460732 19924 init.cc:377] Logged in from keytab as oryx/127.12.248.125@KRBTEST.COM (short username oryx)
I20260925 10:26:02.462273 19924 webserver.cc:460] Webserver started at http://127.12.248.125:37021/ using document root <none> and password file <none>
I20260925 10:26:02.463048 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:02.463135 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:02.463394 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:02.466552 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/instance:
uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
format_stamp: "Formatted at 2026-09-25 10:26:02 on dist-test-slave-f9rl"
server_key: "b859a40e0d661490fc6354a9c4801baf"
server_key_iv: "c4f48afd4b6741d66966a0c7bda78b9a"
server_key_version: "encryptionkey@0"
I20260925 10:26:02.467263 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal/instance:
uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
format_stamp: "Formatted at 2026-09-25 10:26:02 on dist-test-slave-f9rl"
server_key: "b859a40e0d661490fc6354a9c4801baf"
server_key_iv: "c4f48afd4b6741d66966a0c7bda78b9a"
server_key_version: "encryptionkey@0"
I20260925 10:26:02.472554 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:26:02.475940 19939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:02.477334 19924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:02.477558 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
format_stamp: "Formatted at 2026-09-25 10:26:02 on dist-test-slave-f9rl"
server_key: "b859a40e0d661490fc6354a9c4801baf"
server_key_iv: "c4f48afd4b6741d66966a0c7bda78b9a"
server_key_version: "encryptionkey@0"
I20260925 10:26:02.477777 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:02.508338 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:02.512784 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:02.513022 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:02.525532 19785 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
I20260925 10:26:02.533115 19924 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.125:41103
I20260925 10:26:02.533134 19990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.125:41103 every 8 connection(s)
I20260925 10:26:02.534725 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
I20260925 10:26:02.538525 19991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:02.539716 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 19924
I20260925 10:26:02.548061 19991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331962, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.125@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:02.566035 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.542092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.12.248.125:41103)
0925 10:26:02.542948 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.542964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.543013 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:26:02.544873 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.544896 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.545302 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.545730 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.547609 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.549305 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.550782 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.551924 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.552143 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.556967 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.556975 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.556987 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.557029 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.561017 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.561912 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.561921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.561929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.562105 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.562488 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.562492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.562494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.563272 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.563603 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.564117 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.564565 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":389,"threads_started":1}
I20260925 10:26:02.569052 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.550917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:41044)
0925 10:26:02.551803 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.551844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.552034 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.552501 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.552510 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.553407 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.553834 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.553850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.555044 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.555053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.556419 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.556432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.557587 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.557596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.557844 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.558442 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.558474 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.561522 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.565377 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.565387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.565393 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.565934 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.567291 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.567297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.567300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.567491 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.568072 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.568079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.568275 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.568475 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.568773 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":514,"threads_started":1}
I20260925 10:26:02.569203 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.550857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.12.248.126:38783)
0925 10:26:02.551424 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.551433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.551949 (+ 516us) server_negotiation.cc:407] Connection header received
0925 10:26:02.552202 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.552208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.552293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.552408 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.554038 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.554882 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.556580 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.557247 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.557357 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.561715 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.561720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.561724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.561779 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.565165 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.566064 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.566069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.566073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.566211 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.567611 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.567614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.567616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.567947 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.568056 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.568575 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.568860 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260925 10:26:02.570950 19991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331962, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.125@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
I20260925 10:26:02.582528 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.571574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:35670)
0925 10:26:02.571743 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.571758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.571841 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.572111 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.572114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.572434 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.572560 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.572567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.573541 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.573544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.574390 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.574400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.575028 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.575032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.575151 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.575730 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.575746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.577752 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.580320 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.580324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.580326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.580669 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.581056 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.581059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.581061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.581181 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.581719 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.581722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.581849 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.581959 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.582248 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:26:02.582716 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.571760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.12.248.125:41103)
0925 10:26:02.571900 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.571904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.571917 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:26:02.571953 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.571956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.572005 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.572078 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.572670 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.573363 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.574504 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.574906 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.574998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.577898 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.577903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.577906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.577931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.580183 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.580780 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.580784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.580786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.580910 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.581271 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.581274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.581276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.581606 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.581709 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.582076 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.582371 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":51}
I20260925 10:26:02.587370 19991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Bootstrap starting.
I20260925 10:26:02.590438 19991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:02.591471 19991 log.cc:824] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:02.594183 19991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: No bootstrap required, opened a new log
I20260925 10:26:02.601293 19991 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } }
I20260925 10:26:02.601716 19991 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:02.601799 19991 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4ee7b2a2974ec29d0a076103cfefe9, State: Initialized, Role: FOLLOWER
I20260925 10:26:02.602332 19991 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } }
I20260925 10:26:02.603127 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } } }
I20260925 10:26:02.603503 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:02.604074 19991 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:26:02.613059 20009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:26:02.613240 20009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:26:02.615490 19924 master_runner.cc:428] Detected that this master bc4ee7b2a2974ec29d0a076103cfefe9 is joining an existing cluster
I20260925 10:26:02.615613 19924 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.248.125:41103
I20260925 10:26:02.641253 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.620031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.12.248.125:41103)
0925 10:26:02.620208 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.620214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.624157 (+ 3943us) server_negotiation.cc:407] Connection header received
0925 10:26:02.624358 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.624364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.624434 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.624492 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.626201 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.627324 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.629810 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.630600 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.630699 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.633938 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.633947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.633954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.633993 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.637935 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.638865 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.638872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.638879 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.639000 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.639411 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.639419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.639426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.639850 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.640011 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.640553 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.640812 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:26:02.641343 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.619464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.12.248.126:38783)
0925 10:26:02.619677 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.619683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.623686 (+ 4003us) server_negotiation.cc:407] Connection header received
0925 10:26:02.624172 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.624178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.624276 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.624356 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.626038 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.627142 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.629749 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.630398 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.630508 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.633942 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.633948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.633953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.633994 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.638024 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.638980 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.638985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.638989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.639150 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.639641 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.639647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.639651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.640021 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.640164 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.640758 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.641011 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:26:02.642354 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.619603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:35680)
0925 10:26:02.624063 (+ 4460us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.624084 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.624238 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.624700 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.624706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.625548 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.626052 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.626069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.627577 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.627585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.629635 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.629657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.630990 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.630998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.631287 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.632007 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.632035 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.633612 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.638074 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.638080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.638087 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.638753 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.639115 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.639122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.639128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.639302 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.640093 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.640100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.640288 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.641484 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.641779 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4309,"mutex_wait_us":181,"thread_start_us":3673,"threads_started":1}
I20260925 10:26:02.642608 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.619547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:41050)
0925 10:26:02.623567 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.623593 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.623996 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.624583 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.624587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.625338 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.625838 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.625852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.627325 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.627331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.629559 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.629575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.630946 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.630958 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.631167 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.632127 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.632150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.633629 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.638194 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.638200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.638203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.638839 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.639297 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.639302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.639305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.639496 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.640247 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.640251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.640438 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.641721 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.642110 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3879,"mutex_wait_us":126,"spinlock_wait_cycles":87168,"thread_start_us":4159,"threads_started":1}
WARNING: no policy specified for oryx/127.12.248.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.124@KRBTEST.COM" created.
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331960, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.126@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
I20260925 10:26:02.667907 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.649399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.12.248.125:41103)
0925 10:26:02.649608 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.649614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.650151 (+ 537us) server_negotiation.cc:407] Connection header received
0925 10:26:02.650499 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.650503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.650591 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.650715 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.652235 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.653371 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.655437 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.656203 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.656334 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.660350 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.660357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.660362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.660408 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.664233 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.665255 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.665261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.665264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.665468 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.666002 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.666008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.666013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.666411 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.666558 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.667188 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.667431 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 10:26:02.667979 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.649203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:35682)
0925 10:26:02.650023 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.650071 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.650291 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.650811 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.650820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.651723 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.652059 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.652075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.653555 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.653563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.655271 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.655289 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.656390 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.656398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.656672 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.657451 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.657480 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.660146 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.664435 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.664450 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.664461 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.665114 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.665666 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.665674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.665681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.665876 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.666590 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.666602 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.666834 (+ 232us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.667058 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.667475 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":301,"threads_started":1}
I20260925 10:26:02.670804 19462 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.248.125:41103: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
member_type: NON_VOTER
last_known_addr {
host: "127.12.248.125"
port: 41103
}
attrs {
promote: true
}
}
dest_uuid: "110d185250964711b03a7dc16bceaf87"
cas_config_opid_index: -1
I20260925 10:26:02.672467 19462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } }
I20260925 10:26:02.676093 20022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } }
I20260925 10:26:02.676980 19966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.124.keytab.
Entry for principal oryx/127.12.248.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.124.keytab.
I20260925 10:26:02.680625 20022 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.248.125:41103
I20260925 10:26:02.681833 19924 master.cc:537] Master@127.12.248.125:41103 shutting down...
I20260925 10:26:02.682478 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:45493
--webserver_interface=127.12.248.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.124.keytab
--principal=oryx/127.12.248.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:26:02.683015 20021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } }
I20260925 10:26:02.683357 20021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:02.690661 19966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 10:26:02.692059 20021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:26:02.695289 19448 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 10:26:02.697095 19924 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:02.697633 19924 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:02.697970 19924 tablet_replica.cc:330] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: stopping tablet replica
I20260925 10:26:02.704669 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.704116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:35690)
0925 10:26:02.704324 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.704495 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:26:02.705018 19448 consensus_peers.cc:487] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 -> Peer bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103): Couldn't send request to peer bc4ee7b2a2974ec29d0a076103cfefe9. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:26:02.715039 19924 master.cc:552] Master@127.12.248.125:41103 shutdown complete.
I20260925 10:26:02.715137 19924 master_runner.cc:305] Clearing existing system tablet
I20260925 10:26:02.715579 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:02.715703 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:02.718899 19924 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:26:02.720538 20027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:02.720916 19924 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:02.721177 19924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:26:02.721273 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
format_stamp: "Formatted at 2026-09-25 10:26:02 on dist-test-slave-f9rl"
server_key: "b859a40e0d661490fc6354a9c4801baf"
server_key_iv: "c4f48afd4b6741d66966a0c7bda78b9a"
server_key_version: "encryptionkey@0"
I20260925 10:26:02.726471 19924 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:02.733536 19924 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:26:02.733739 19924 log.cc:1196] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:26:02.734205 19924 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Deleting consensus metadata
I20260925 10:26:02.734478 19924 master_runner.cc:315] Copying system tablet from 127.12.248.126:38783
I20260925 10:26:02.743218 19924 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.126:38783
I20260925 10:26:02.761937 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.744319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.12.248.126:38783)
0925 10:26:02.744511 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:02.744518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:02.747703 (+ 3185us) server_negotiation.cc:407] Connection header received
0925 10:26:02.747900 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:02.747906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:02.748034 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:02.748150 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:02.749310 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.750454 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.752326 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.752981 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.753124 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.755483 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:02.755490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:02.755495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:02.755534 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:02.758934 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.759736 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.759741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.759745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.759892 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:02.760310 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:02.760314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:02.760317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:02.760675 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:02.760825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:02.761303 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:26:02.761582 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:26:02.762019 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:02.744202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:41052)
0925 10:26:02.747551 (+ 3349us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:02.747578 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:02.747753 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:02.748202 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:02.748207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:02.748769 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:02.749057 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.749070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.750630 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.750637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.752152 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:02.752168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:02.753151 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:02.753157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:02.753346 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:02.754117 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:02.754141 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:02.755229 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:02.759078 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.759084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.759088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.759605 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.760018 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:02.760024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:02.760028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:02.760194 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:02.760799 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:02.760805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:02.760997 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 10:26:02.761159 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 10:26:02.761554 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3184,"thread_start_us":3009,"threads_started":1}
I20260925 10:26:02.763406 19482 tablet_copy_service.cc:140] P 110d185250964711b03a7dc16bceaf87: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bc4ee7b2a2974ec29d0a076103cfefe9 ({username='oryx', principal='oryx/127.12.248.125@KRBTEST.COM'} at 127.0.0.1:41052)
I20260925 10:26:02.763690 19482 tablet_copy_service.cc:161] P 110d185250964711b03a7dc16bceaf87: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bc4ee7b2a2974ec29d0a076103cfefe9 at {username='oryx', principal='oryx/127.12.248.125@KRBTEST.COM'} at 127.0.0.1:41052: session id = bc4ee7b2a2974ec29d0a076103cfefe9-00000000000000000000000000000000
I20260925 10:26:02.768769 19482 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:26:02.771564 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:26:02.779528 19924 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: tablet copy: Starting download of 0 data blocks...
I20260925 10:26:02.779886 19924 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: tablet copy: Starting download of 1 WAL segments...
I20260925 10:26:02.783866 19924 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:26:02.789546 19482 tablet_copy_service.cc:342] P 110d185250964711b03a7dc16bceaf87: Request end of tablet copy session bc4ee7b2a2974ec29d0a076103cfefe9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.248.125@KRBTEST.COM'} at 127.0.0.1:41052
I20260925 10:26:02.789810 19482 tablet_copy_service.cc:433] P 110d185250964711b03a7dc16bceaf87: ending tablet copy session bc4ee7b2a2974ec29d0a076103cfefe9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bc4ee7b2a2974ec29d0a076103cfefe9
I20260925 10:26:02.804715 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:02.807816 20034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:02.808683 20035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:02.813922 19924 server_base.cc:1034] running on GCE node
W20260925 10:26:02.814196 20037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:02.814626 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:02.815316 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:02.816572 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790331962816462 us; error 39 us; skew 500 ppm
I20260925 10:26:02.816886 19924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:02 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331962, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:02.819465 19924 init.cc:377] Logged in from keytab as oryx/127.12.248.125@KRBTEST.COM (short username oryx)
I20260925 10:26:02.820888 19924 webserver.cc:460] Webserver started at http://127.12.248.125:41653/ using document root <none> and password file <none>
I20260925 10:26:02.821306 19924 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:02.821404 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:02.824826 19924 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 10:26:02.827008 20044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:02.827500 19924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:26:02.827653 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
format_stamp: "Formatted at 2026-09-25 10:26:02 on dist-test-slave-f9rl"
server_key: "b859a40e0d661490fc6354a9c4801baf"
server_key_iv: "c4f48afd4b6741d66966a0c7bda78b9a"
server_key_version: "encryptionkey@0"
I20260925 10:26:02.827826 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:02.862673 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:02.867992 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:02.889683 19924 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.125:41103
I20260925 10:26:02.889707 20095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.125:41103 every 8 connection(s)
I20260925 10:26:02.890815 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
I20260925 10:26:02.896627 20096 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:26:02.901177 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Bootstrap starting.
I20260925 10:26:02.919834 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:26:02.920439 20096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: Bootstrap complete.
I20260925 10:26:02.921609 20096 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } }
I20260925 10:26:02.921903 20096 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc4ee7b2a2974ec29d0a076103cfefe9, State: Initialized, Role: LEARNER
I20260925 10:26:02.922184 20096 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } }
I20260925 10:26:02.923202 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } }
I20260925 10:26:02.923662 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:26:02.923512 20096 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:02.932319 20110 catalog_manager.cc:1259] Loaded cluster ID: 4d36fe76d3444ea7a64880ca0becff65
I20260925 10:26:02.932435 20110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: loading cluster ID for follower catalog manager: success
I20260925 10:26:02.936302 20110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: acquiring CA information for follower catalog manager: success
I20260925 10:26:02.938880 20110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:26:03.032467 19919 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
W20260925 10:26:03.138224 20024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:03.138583 20024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:03.138741 20024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:03.146533 20024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:03.146677 20024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:03.146786 20024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:03.146890 20024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:03.146966 20024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:03.158197 20024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.124.keytab
--principal=oryx/127.12.248.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.248.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:03.159659 20024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:03.161052 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:03.173730 20116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:03.173782 20118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:03.173734 20115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:03.174744 20024 server_base.cc:1034] running on GCE node
I20260925 10:26:03.175496 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:03.176990 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:03.178283 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790331963178237 us; error 45 us; skew 500 ppm
I20260925 10:26:03.178658 20024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331963, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:03.182240 20024 init.cc:377] Logged in from keytab as oryx/127.12.248.124@KRBTEST.COM (short username oryx)
I20260925 10:26:03.183885 20024 webserver.cc:460] Webserver started at http://127.12.248.124:46185/ using document root <none> and password file <none>
I20260925 10:26:03.184652 20024 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:03.184834 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:03.185151 20024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:03.190012 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/instance:
uuid: "f72d5c990ecf403db8800cee4d2e735a"
format_stamp: "Formatted at 2026-09-25 10:26:03 on dist-test-slave-f9rl"
server_key: "666ec7d7aa12cb267268170d2bf2a9f2"
server_key_iv: "72b3cf537953d1837df4c9fc29251c89"
server_key_version: "encryptionkey@0"
I20260925 10:26:03.190836 20024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal/instance:
uuid: "f72d5c990ecf403db8800cee4d2e735a"
format_stamp: "Formatted at 2026-09-25 10:26:03 on dist-test-slave-f9rl"
server_key: "666ec7d7aa12cb267268170d2bf2a9f2"
server_key_iv: "72b3cf537953d1837df4c9fc29251c89"
server_key_version: "encryptionkey@0"
I20260925 10:26:03.196750 20024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 10:26:03.200510 20125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:03.202040 20024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:03.202234 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "f72d5c990ecf403db8800cee4d2e735a"
format_stamp: "Formatted at 2026-09-25 10:26:03 on dist-test-slave-f9rl"
server_key: "666ec7d7aa12cb267268170d2bf2a9f2"
server_key_iv: "72b3cf537953d1837df4c9fc29251c89"
server_key_version: "encryptionkey@0"
I20260925 10:26:03.202443 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:03.237592 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:03.242707 20024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:03.243144 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:03.269946 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.261289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:33382)
0925 10:26:03.262179 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.262202 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.262353 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.262869 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.262874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.262904 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:03.263098 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.263107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.265192 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.265197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.267573 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.267587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.268893 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.268900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.269077 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.269137 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.269267 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.269504 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":639,"threads_started":1}
I20260925 10:26:03.270291 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.261460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.12.248.125:41103)
0925 10:26:03.262516 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.262521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.262538 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:03.262598 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.262602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.262671 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.262823 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:03.263255 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.264978 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.267780 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.268688 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.268893 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.269100 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.269300 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.269504 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":809,"threads_started":1}
I20260925 10:26:03.272053 20071 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 10:26:03.272818 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110d185250964711b03a7dc16bceaf87. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } }
I20260925 10:26:03.273128 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:26:03.274055 20024 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.124:45493
I20260925 10:26:03.274063 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.124:45493 every 8 connection(s)
I20260925 10:26:03.275628 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
I20260925 10:26:03.279661 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:03.286126 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20024
I20260925 10:26:03.287055 20180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331963, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.124@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
I20260925 10:26:03.311695 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.290352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.12.248.126:38783)
0925 10:26:03.290948 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.290955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.291844 (+ 889us) server_negotiation.cc:407] Connection header received
0925 10:26:03.293587 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.293593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.293685 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.293805 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.295581 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.296386 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.298631 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.299360 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.299520 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.303729 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.303737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.303743 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.303805 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.307731 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.308770 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.308774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.308777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.308899 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.309514 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.309524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.309527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.309809 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.309951 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.310984 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.311223 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":299,"threads_started":1}
I20260925 10:26:03.312510 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.290428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:48918)
0925 10:26:03.291685 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.291737 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.293292 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.293921 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.293964 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.294777 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.295389 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.295407 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.296572 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.296576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.298452 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.298485 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.299605 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.299610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.299804 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.300549 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.300577 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.303444 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.307878 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.307892 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.307907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.308653 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.309037 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.309044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.309048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.309325 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.309950 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.309959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.310472 (+ 513us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.310779 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.311205 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":769,"threads_started":1}
I20260925 10:26:03.313747 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.288453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.12.248.124:45493)
0925 10:26:03.289374 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.289385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.289427 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:26:03.293275 (+ 3848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.293298 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.293664 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.296401 (+ 2737us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.296638 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.298618 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.300326 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.300948 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.301050 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.306713 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.306723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.306738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.306770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.310487 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.311358 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.311366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.311368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.311528 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.312010 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.312014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.312016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.312369 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.312739 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.313047 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.313287 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":469,"threads_started":1}
I20260925 10:26:03.314425 20180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331963, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.124@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
I20260925 10:26:03.331151 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.315608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:33396)
0925 10:26:03.315877 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.315892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.315976 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.316330 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.316338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.316799 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.317094 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.317104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.318286 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.318290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.319232 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.319245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.320104 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.320109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.320290 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.320986 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.321002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.323074 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.327781 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.327787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.327791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.328573 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.329079 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.329083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.329085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.329497 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.330210 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.330214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.330362 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.330531 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.330876 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 10:26:03.331430 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.315584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.12.248.125:41103)
0925 10:26:03.315801 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.315809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.315958 (+ 149us) server_negotiation.cc:407] Connection header received
0925 10:26:03.316081 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.316086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.316165 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.316280 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.317227 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.318164 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.319356 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.319951 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.320048 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.323277 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.323283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.323287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.323324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.327587 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.328700 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.328706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.328710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.328950 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.329618 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.329623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.329626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.330050 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.330210 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.330643 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.330984 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:26:03.332528 20180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331963, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.124@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
I20260925 10:26:03.346279 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.333247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44504)
0925 10:26:03.333478 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.333499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.333592 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.333862 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.333865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.334198 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.334340 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.334347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.335697 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.335701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.336598 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.336609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.337618 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.337623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.337882 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.338620 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.338637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.340843 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.343799 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.343803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.343805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.344299 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.344626 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.344630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.344632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.344873 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.345417 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.345421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.345647 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.345765 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.345973 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 10:26:03.346405 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.333296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.12.248.124:45493)
0925 10:26:03.333483 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.333547 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.333567 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:26:03.333690 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.333695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.333764 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.333856 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.334464 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.335566 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.336733 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.337403 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.337564 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.341027 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.341032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.341035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.341064 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.343655 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.344406 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.344410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.344413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.344525 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.344968 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.344971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.344973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.345308 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.345416 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.345864 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.346097 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":58}
I20260925 10:26:03.351428 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Bootstrap starting.
I20260925 10:26:03.354822 20180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:03.355865 20180 log.cc:824] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:03.359176 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: No bootstrap required, opened a new log
I20260925 10:26:03.368280 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } }
I20260925 10:26:03.368695 20180 raft_consensus.cc:374] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:03.368830 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72d5c990ecf403db8800cee4d2e735a, State: Initialized, Role: FOLLOWER
I20260925 10:26:03.369396 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } }
I20260925 10:26:03.370594 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } } }
I20260925 10:26:03.371279 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:03.371515 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:26:03.381929 20198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:26:03.382091 20198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:26:03.386461 20024 master_runner.cc:428] Detected that this master f72d5c990ecf403db8800cee4d2e735a is joining an existing cluster
I20260925 10:26:03.386631 20024 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.248.124:45493
I20260925 10:26:03.418303 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.392466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.12.248.124:45493)
0925 10:26:03.392769 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.392777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.397916 (+ 5139us) server_negotiation.cc:407] Connection header received
0925 10:26:03.398170 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.398174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.398289 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.398396 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.400149 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.401719 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.404025 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.404969 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.405083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.408397 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.408403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.408408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.408458 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.414165 (+ 5707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415446 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.415451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.415454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.415769 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.416247 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.416254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.416260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.416732 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.416882 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.417536 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.417812 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":64}
I20260925 10:26:03.418319 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.391047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.12.248.126:38783)
0925 10:26:03.391249 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.391257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.392121 (+ 864us) server_negotiation.cc:407] Connection header received
0925 10:26:03.392456 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.392463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.392623 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.392757 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.398153 (+ 5396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.399477 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.402715 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.403631 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.403762 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.407649 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.407658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.407664 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.407718 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.412742 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.414760 (+ 2018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.414769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.414775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.414995 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415739 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.415745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.415749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.416289 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.416515 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.417507 (+ 992us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.417837 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:26:03.418522 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.390893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.12.248.125:41103)
0925 10:26:03.391552 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.391562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.393768 (+ 2206us) server_negotiation.cc:407] Connection header received
0925 10:26:03.396081 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.396088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.396190 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.396279 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.398066 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.399681 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.405165 (+ 5484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.405846 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.405970 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.409848 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.409913 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.409920 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.409962 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.414214 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415430 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.415436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.415439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.415712 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.416288 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.416294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.416297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.416740 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.416867 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.417632 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.417953 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501}
I20260925 10:26:03.419775 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.392105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44508)
0925 10:26:03.397809 (+ 5704us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.397823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.397978 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.398627 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.398632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.399430 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.399945 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.399958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.401943 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.401951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.403863 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.403901 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.405370 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.405381 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.405615 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.406352 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.406393 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.408187 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.414322 (+ 6135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.414329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.414333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.415334 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.415888 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.415894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.416129 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.416967 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.416970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.417207 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.418729 (+ 1522us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.419030 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5587,"mutex_wait_us":41,"spinlock_wait_cycles":66560,"thread_start_us":5404,"threads_started":1}
I20260925 10:26:03.420987 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.391064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:48920)
0925 10:26:03.391975 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.392001 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.392295 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.393896 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.393905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.395173 (+ 1268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.395630 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.395662 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.399772 (+ 4110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.399779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.402499 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.402516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.404124 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.404138 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.404384 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.405546 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.405576 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.407326 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.412966 (+ 5640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.412973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.412978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.414584 (+ 1606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.415201 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.415211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.415557 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.416638 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.416645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.417023 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.419143 (+ 2120us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.419647 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":1,"spinlock_wait_cycles":60800,"thread_start_us":510,"threads_started":1}
I20260925 10:26:03.421490 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19523
I20260925 10:26:03.421131 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.391425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:33402)
0925 10:26:03.393620 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.393643 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.395869 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.396522 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.396528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.397363 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.397884 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.397907 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.399920 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.399925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.404981 (+ 5056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.404990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.406429 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.406439 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.406718 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.407538 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.407560 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.409549 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.414389 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.414408 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.414412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.415267 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.415864 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.415871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.415876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.416168 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.416950 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.416955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.417308 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.418696 (+ 1388us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.420406 (+ 1710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2047,"mutex_wait_us":3384,"thread_start_us":1787,"threads_started":1}
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331960, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.126@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.65@KRBTEST.COM
I20260925 10:26:03.446316 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.428177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44522)
0925 10:26:03.428391 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.428411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.428538 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.429702 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.429713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.430278 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.430607 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.430618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.431623 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.431631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.433593 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.433609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.434766 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.434772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.434959 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.435594 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.435633 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.439215 (+ 3582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.443003 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.443010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.443015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.443623 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.444179 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.444186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.444191 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.444369 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.445099 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.445105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.445291 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.445602 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.445927 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:26:03.447352 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.429132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.12.248.124:45493)
0925 10:26:03.429294 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.429298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.429312 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:26:03.429383 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.429386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.429534 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.429653 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.430779 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.431485 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.433757 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.434592 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.434741 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.439473 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.439481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.439486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.439546 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.442832 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.443807 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.443813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.443817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.443984 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.444505 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.444512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.444516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.444959 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.445142 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.446786 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.447020 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:26:03.448962 19462 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.248.124:45493: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a"
member_type: NON_VOTER
last_known_addr {
host: "127.12.248.124"
port: 45493
}
attrs {
promote: true
}
}
dest_uuid: "110d185250964711b03a7dc16bceaf87"
cas_config_opid_index: 5
I20260925 10:26:03.450071 19462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } }
I20260925 10:26:03.451658 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.423507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.12.248.65:39655)
0925 10:26:03.424826 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.424838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.424874 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:26:03.424958 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.424970 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.425382 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.429380 (+ 3998us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.429830 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.431708 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.433830 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.435081 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.435268 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.442910 (+ 7642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.442933 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.442956 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.443000 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.446994 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.448634 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.448646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.448651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.448833 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.449382 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.449388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.449392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.449953 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.450348 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.450819 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.451107 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":1802,"threads_started":1}
I20260925 10:26:03.452646 20155 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f72d5c990ecf403db8800cee4d2e735a (127.12.248.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.454474 20071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:26:03.457648 20152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:03.458014 19646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:03.458046 20155 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.248.124:45493
I20260925 10:26:03.458841 20024 master.cc:537] Master@127.12.248.124:45493 shutting down...
I20260925 10:26:03.459836 20155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:26:03.462706 20152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:26:03.464121 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.464478 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:03.468896 20208 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:03.479529 20071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f72d5c990ecf403db8800cee4d2e735a (127.12.248.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.486450 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: NON_VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: true } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.486840 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:26:03.488080 20209 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: attempting to promote NON_VOTER bc4ee7b2a2974ec29d0a076103cfefe9 to VOTER
I20260925 10:26:03.488988 20209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } }
I20260925 10:26:03.491900 20071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 LEARNER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:26:03.492623 20155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:26:03.495517 20208 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.499753 20208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.500038 20208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:03.500720 20208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.500983 20208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:03.501083 20071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
W20260925 10:26:03.502727 20131 tablet.cc:2369] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:03.505553 20187 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:03.505911 20187 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } }
I20260925 10:26:03.508129 20187 leader_election.cc:288] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103)
I20260925 10:26:03.508773 20070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" is_pre_election: true
I20260925 10:26:03.508888 19472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110d185250964711b03a7dc16bceaf87" is_pre_election: true
I20260925 10:26:03.514084 20127 leader_election.cc:302] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f72d5c990ecf403db8800cee4d2e735a; no voters: 110d185250964711b03a7dc16bceaf87, bc4ee7b2a2974ec29d0a076103cfefe9
I20260925 10:26:03.514544 20187 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:26:03.515323 20024 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:03.515722 20024 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:03.515967 20024 tablet_replica.cc:330] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: stopping tablet replica
I20260925 10:26:03.523377 20024 master.cc:552] Master@127.12.248.124:45493 shutdown complete.
I20260925 10:26:03.523515 20024 master_runner.cc:305] Clearing existing system tablet
I20260925 10:26:03.523435 20099 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.523720 20099 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:03.524632 20024 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:03.524792 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:03.530174 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.529404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44538)
0925 10:26:03.529662 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.529818 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 10:26:03.530406 19447 consensus_peers.cc:487] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 -> Peer f72d5c990ecf403db8800cee4d2e735a (127.12.248.124:45493): Couldn't send request to peer f72d5c990ecf403db8800cee4d2e735a. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45493: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:26:03.531687 20024 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.002s
I20260925 10:26:03.533739 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:03.534257 20024 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:03.534500 20024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 10:26:03.534600 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "f72d5c990ecf403db8800cee4d2e735a"
format_stamp: "Formatted at 2026-09-25 10:26:03 on dist-test-slave-f9rl"
server_key: "666ec7d7aa12cb267268170d2bf2a9f2"
server_key_iv: "72b3cf537953d1837df4c9fc29251c89"
server_key_version: "encryptionkey@0"
I20260925 10:26:03.540004 20024 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:03.547065 20024 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:26:03.547390 20024 log.cc:1196] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:26:03.548211 20024 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Deleting consensus metadata
I20260925 10:26:03.548630 20024 master_runner.cc:315] Copying system tablet from 127.12.248.126:38783
I20260925 10:26:03.551548 20024 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: tablet copy: Beginning tablet copy session from remote peer at address 127.12.248.126:38783
I20260925 10:26:03.569033 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.552463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:48934)
0925 10:26:03.553403 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:03.553423 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:03.553641 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:03.554105 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:03.554111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:03.554618 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:03.554866 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.554879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.556437 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.556443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.558088 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:03.558104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.559223 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.559230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:03.559400 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.560161 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:03.560185 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:03.561303 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:03.565616 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.565623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.565627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.566218 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.566695 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:03.566700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:03.566703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:03.566917 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:03.567597 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:03.567604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:03.567841 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 10:26:03.568023 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 10:26:03.568379 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":634,"threads_started":1}
I20260925 10:26:03.569046 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:03.552523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.12.248.126:38783)
0925 10:26:03.552759 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:03.552766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:03.553585 (+ 819us) server_negotiation.cc:407] Connection header received
0925 10:26:03.553827 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:03.553835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:03.553931 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:03.554046 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:03.555054 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.556260 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.558259 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:03.559044 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:03.559188 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:03.561569 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:03.561576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:03.561582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:03.561628 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:03.565473 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.566365 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.566371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.566375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.566560 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:03.567048 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:03.567054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:03.567058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:03.567450 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:03.567627 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:03.568173 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 10:26:03.568410 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 10:26:03.570278 19482 tablet_copy_service.cc:140] P 110d185250964711b03a7dc16bceaf87: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f72d5c990ecf403db8800cee4d2e735a ({username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:48934)
I20260925 10:26:03.570464 19482 tablet_copy_service.cc:161] P 110d185250964711b03a7dc16bceaf87: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f72d5c990ecf403db8800cee4d2e735a at {username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:48934: session id = f72d5c990ecf403db8800cee4d2e735a-00000000000000000000000000000000
I20260925 10:26:03.574299 19482 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:26:03.576538 20024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:26:03.585350 20024 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: tablet copy: Starting download of 0 data blocks...
I20260925 10:26:03.585767 20024 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: tablet copy: Starting download of 1 WAL segments...
I20260925 10:26:03.588918 20024 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:26:03.593664 19482 tablet_copy_service.cc:342] P 110d185250964711b03a7dc16bceaf87: Request end of tablet copy session f72d5c990ecf403db8800cee4d2e735a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:48934
I20260925 10:26:03.593830 19482 tablet_copy_service.cc:433] P 110d185250964711b03a7dc16bceaf87: ending tablet copy session f72d5c990ecf403db8800cee4d2e735a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f72d5c990ecf403db8800cee4d2e735a
I20260925 10:26:03.598215 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19523
I20260925 10:26:03.607534 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:03.611234 20222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:03.615047 20223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:03.625335 20024 server_base.cc:1034] running on GCE node
I20260925 10:26:03.626544 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:39655
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:38783
--webserver_port=33927
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:03.626778 20225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:03.627295 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:03.628185 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:03.629385 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790331963629330 us; error 37 us; skew 500 ppm
I20260925 10:26:03.629703 20024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:03 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331963, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:03.631942 20024 init.cc:377] Logged in from keytab as oryx/127.12.248.124@KRBTEST.COM (short username oryx)
I20260925 10:26:03.637799 20024 webserver.cc:460] Webserver started at http://127.12.248.124:36805/ using document root <none> and password file <none>
I20260925 10:26:03.638198 20024 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:03.638315 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:03.641785 20024 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260925 10:26:03.644002 20233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:03.644533 20024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 10:26:03.644721 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "f72d5c990ecf403db8800cee4d2e735a"
format_stamp: "Formatted at 2026-09-25 10:26:03 on dist-test-slave-f9rl"
server_key: "666ec7d7aa12cb267268170d2bf2a9f2"
server_key_iv: "72b3cf537953d1837df4c9fc29251c89"
server_key_version: "encryptionkey@0"
I20260925 10:26:03.645002 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:03.675503 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:03.683305 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:03.709398 20024 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.124:45493
I20260925 10:26:03.709501 20284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.124:45493 every 8 connection(s)
I20260925 10:26:03.710757 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
I20260925 10:26:03.716513 20285 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:26:03.720489 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Bootstrap starting.
I20260925 10:26:03.740487 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:26:03.741070 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Bootstrap complete.
I20260925 10:26:03.742254 20285 raft_consensus.cc:348] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } }
I20260925 10:26:03.742552 20285 raft_consensus.cc:729] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f72d5c990ecf403db8800cee4d2e735a, State: Initialized, Role: LEARNER
I20260925 10:26:03.742827 20285 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } }
I20260925 10:26:03.743973 20285 sys_catalog.cc:564] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:03.743728 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:03.744258 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: LEARNER
I20260925 10:26:03.753532 20299 catalog_manager.cc:1259] Loaded cluster ID: 4d36fe76d3444ea7a64880ca0becff65
I20260925 10:26:03.753647 20299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: loading cluster ID for follower catalog manager: success
I20260925 10:26:03.756984 20299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: acquiring CA information for follower catalog manager: success
I20260925 10:26:03.759501 20299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:26:04.014206 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:04.014634 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:04.014817 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:04.023738 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:04.023862 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:04.024030 20229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:04.037886 20229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:39655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=33927
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:04.039302 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:04.041009 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:26:04.042320 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.034620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44554)
0925 10:26:04.035088 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.035111 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.035269 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.036048 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.036052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.036074 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:04.036479 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.036496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.038477 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.038484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.039722 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.039734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.041351 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.041357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.041582 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.041617 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.041725 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.041889 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":248,"threads_started":1}
I20260925 10:26:04.042587 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.034780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.12.248.124:45493)
0925 10:26:04.035416 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.035422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.035446 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:26:04.035527 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.035532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.035628 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.035763 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:04.036677 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.038321 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.039862 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.041149 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.041347 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.041612 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.041761 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.041937 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":743,"threads_started":1}
I20260925 10:26:04.043814 20260 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 10:26:04.044495 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110d185250964711b03a7dc16bceaf87. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:04.044795 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: LEARNER
W20260925 10:26:04.058903 20306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:04.059314 20307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:04.058910 20309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:04.061086 20229 server_base.cc:1034] running on GCE node
I20260925 10:26:04.061774 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:04.062901 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:04.064169 20229 hybrid_clock.cc:648] HybridClock initialized: now 1790331964064054 us; error 39 us; skew 500 ppm
I20260925 10:26:04.064687 20229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:04.068217 20229 init.cc:377] Logged in from keytab as oryx/127.12.248.65@KRBTEST.COM (short username oryx)
I20260925 10:26:04.070135 20229 webserver.cc:460] Webserver started at http://127.12.248.65:33927/ using document root <none> and password file <none>
I20260925 10:26:04.070935 20229 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:04.071048 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:04.076726 20229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 10:26:04.079854 20316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.081223 20229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:04.081362 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "69421f70a24d423387c1b7f7d8db5e42"
format_stamp: "Formatted at 2026-09-25 10:26:00 on dist-test-slave-f9rl"
server_key: "acfc9078d9fa4f04d2fe285d0434bdc3"
server_key_iv: "3d26ba06d3fdee20fbfe527e74b50db0"
server_key_version: "encryptionkey@0"
I20260925 10:26:04.082394 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:04.119990 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:04.125422 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:04.125818 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:04.127074 20229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:04.128831 20229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:04.128938 20229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.129086 20229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:04.129179 20229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.173641 20229 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:39655
I20260925 10:26:04.173812 20428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:39655 every 8 connection(s)
I20260925 10:26:04.175220 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:04.183465 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20229
I20260925 10:26:04.183791 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19657
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.66@KRBTEST.COM
I20260925 10:26:04.219439 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.179582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:59195 (local address 127.12.248.124:45493)
0925 10:26:04.179807 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.179812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.187231 (+ 7419us) server_negotiation.cc:407] Connection header received
0925 10:26:04.187642 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.187652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.187759 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.187883 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.194741 (+ 6858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.195954 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.198313 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.199141 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.199248 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.206011 (+ 6763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.206016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.206019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.206053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.211061 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.212392 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.212399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.212402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.212556 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.213104 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.213110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.213113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.213502 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.213634 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.218589 (+ 4955us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.218977 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:26:04.220033 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.185228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.12.248.66:44403)
0925 10:26:04.195209 (+ 9981us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.195216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.195241 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:26:04.195295 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.195301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.195491 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.195688 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.197780 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.199373 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.201486 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.202487 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.205566 (+ 3079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.212132 (+ 6566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.212156 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.212178 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.212231 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.216101 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.217065 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.217075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.217079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.217228 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.217756 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.217762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.217765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.218230 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.218566 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.218974 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.219274 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9781,"thread_start_us":9619,"threads_started":1}
I20260925 10:26:04.221766 19780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:04.222041 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.180886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:60921 (local address 127.12.248.125:41103)
0925 10:26:04.181892 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.181898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.181928 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:26:04.184673 (+ 2745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.184683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.184825 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.184980 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.188353 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.190570 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.198077 (+ 7507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.198760 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.198891 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.207480 (+ 8589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.207488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.207493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.207552 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.212209 (+ 4657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.212994 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.213001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.213007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.213175 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.216926 (+ 3751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.216935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.216939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.217571 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.217785 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.221344 (+ 3559us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.221608 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":1010,"threads_started":1}
I20260925 10:26:04.222751 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.187263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:38161 (local address 127.12.248.126:38783)
0925 10:26:04.194325 (+ 7062us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.194334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.194970 (+ 636us) server_negotiation.cc:407] Connection header received
0925 10:26:04.195044 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.195050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.195147 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.195259 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.196646 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.197947 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.200644 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.201560 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.201773 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.207546 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.207555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.207571 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.207649 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.213603 (+ 5954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.219655 (+ 6052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.219661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.219665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.219861 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.220415 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.220421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.220425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.220825 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.221009 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.222058 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.222491 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6892,"thread_start_us":6823,"threads_started":1}
I20260925 10:26:04.221858 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.181605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.12.248.65:59195)
0925 10:26:04.187107 (+ 5502us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.187129 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.187355 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.188146 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.188154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.192518 (+ 4364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.193094 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.193114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.196165 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.196170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.198120 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.198135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.199545 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.199551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.199851 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.201796 (+ 1945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.201819 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.205676 (+ 3857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.211305 (+ 5629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.211315 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.211335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.212254 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.212721 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.212728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.212731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.212979 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.213727 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.213735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.217962 (+ 4227us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x60c000034ab8. stack: 00007f461018e66e 00007f461018e430 00007f460f2155ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f46101840fd 00007f4610183d1e 00007f4616c9b664 00007f4616c9ce4d 00007f4616b9cf9a 00007f4616b97644 00007f4616b9233a 00007f4616c19dca 00007f4616c17f68 00007f4616c4b82f
0925 10:26:04.218053 (+ 91us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.219733 (+ 1680us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.220049 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5224,"mutex_wait_us":89,"spinlock_wait_cycles":6042880,"thread_start_us":4972,"threads_started":1}
I20260925 10:26:04.223387 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.180042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.12.248.65:60921)
0925 10:26:04.181637 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.181707 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.184296 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.185678 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.185696 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.186765 (+ 1069us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.188028 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.188057 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.193356 (+ 5299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.193365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.197876 (+ 4511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.197890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.199198 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.199205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.199525 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.200453 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.200497 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.207119 (+ 6622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.212379 (+ 5260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.212383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.212386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.212855 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.213309 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.213313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.213316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.216763 (+ 3447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.217731 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.217738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.218024 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.219131 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.219730 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":67,"thread_start_us":1832,"threads_started":1}
I20260925 10:26:04.223726 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.188388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:38161)
0925 10:26:04.194701 (+ 6313us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.194723 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.194859 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.195441 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.195446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.195989 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.196444 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.196457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.198147 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.198152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.199906 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.200478 (+ 572us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.202044 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.202063 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.202327 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.203030 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.203051 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.206810 (+ 3759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.213822 (+ 7012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.213828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.213833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.218297 (+ 4464us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x60c000034ab8. stack: 00007f461018e66e 00007f461018e430 00007f460f2155ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f46101840fd 00007f4610183d1e 00007f4616c9b664 00007f4616b9ded8 00007f4616b9c524 00007f4616b97575 00007f4616b9233a 00007f4616c19dca 00007f4616c17f68 00007f4616c4b82f
0925 10:26:04.218875 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.220036 (+ 1161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.220049 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.220053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.220246 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.220995 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.221000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.221160 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.221297 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.221595 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6173,"mutex_wait_us":37,"spinlock_wait_cycles":6659456,"thread_start_us":5972,"threads_started":1}
I20260925 10:26:04.225206 20429 heartbeater.cc:347] Connected to a master server at 127.12.248.124:45493
I20260925 10:26:04.225342 20430 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:04.225592 20430 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.225648 20431 heartbeater.cc:347] Connected to a master server at 127.12.248.125:41103
I20260925 10:26:04.225860 20431 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.225597 20429 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.226944 20430 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
I20260925 10:26:04.227129 20431 heartbeater.cc:511] Master 127.12.248.125:41103 requested a full tablet report, sending...
I20260925 10:26:04.227800 20429 heartbeater.cc:511] Master 127.12.248.124:45493 requested a full tablet report, sending...
I20260925 10:26:04.229038 19462 ts_manager.cc:194] Re-registered known tserver with Master: 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655)
I20260925 10:26:04.230064 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:38161
I20260925 10:26:04.230293 20061 ts_manager.cc:194] Registered new tserver with Master: 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655)
I20260925 10:26:04.231647 20250 ts_manager.cc:194] Registered new tserver with Master: 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655)
I20260925 10:26:04.316041 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19657
I20260925 10:26:04.333618 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:44403
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:38783
--webserver_port=36453
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:26:04.492437 20209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: NON_VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: true } } }
I20260925 10:26:04.492873 20209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:04.494704 20442 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: attempting to promote NON_VOTER f72d5c990ecf403db8800cee4d2e735a to VOTER
I20260925 10:26:04.495318 20442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:04.497215 20071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:26:04.497291 20260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 LEARNER]: Refusing update from remote peer 110d185250964711b03a7dc16bceaf87: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 10:26:04.497855 20442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:26:04.498279 20443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 10:26:04.500888 20442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f72d5c990ecf403db8800cee4d2e735a (127.12.248.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.501595 20071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f72d5c990ecf403db8800cee4d2e735a (127.12.248.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.505051 20209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.505934 20209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:04.506328 20209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.506628 20209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:04.511798 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.512050 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:04.513034 20260 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f72d5c990ecf403db8800cee4d2e735a (127.12.248.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.513504 20446 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.514037 20446 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:04.517325 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "110d185250964711b03a7dc16bceaf87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:04.517637 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:26:04.679237 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:04.679528 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:04.679649 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:04.687202 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:04.687355 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:04.687500 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:04.698237 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:44403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=36453
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:04.699534 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:04.700805 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:04.714389 20451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:04.714339 20452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:04.714993 20454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:04.715785 20441 server_base.cc:1034] running on GCE node
I20260925 10:26:04.716410 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:04.717417 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:04.718729 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790331964718663 us; error 52 us; skew 500 ppm
I20260925 10:26:04.719046 20441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:04.722594 20441 init.cc:377] Logged in from keytab as oryx/127.12.248.66@KRBTEST.COM (short username oryx)
I20260925 10:26:04.724367 20441 webserver.cc:460] Webserver started at http://127.12.248.66:36453/ using document root <none> and password file <none>
I20260925 10:26:04.725129 20441 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:04.725296 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:04.731086 20441 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:26:04.733981 20461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.735378 20441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:26:04.735562 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "7178757e204696c2de6cf41f23d5d0a2"
server_key_iv: "5740901d3c2f5ddebaedd8c8b02f550e"
server_key_version: "encryptionkey@0"
I20260925 10:26:04.736599 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:04.762271 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:04.766776 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:04.767071 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:04.768042 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:04.769512 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:04.769629 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.769759 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:04.769853 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:04.804097 20441 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:44403
I20260925 10:26:04.804157 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:44403 every 8 connection(s)
I20260925 10:26:04.805577 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:04.815963 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20441
I20260925 10:26:04.816262 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19791
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331964, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
Sep 25 10:26:04 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.67@KRBTEST.COM
I20260925 10:26:04.834724 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.808560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53329 (local address 127.12.248.126:38783)
0925 10:26:04.809753 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.809760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.810725 (+ 965us) server_negotiation.cc:407] Connection header received
0925 10:26:04.813512 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.813517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.813594 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.813711 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.816731 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.817793 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.820328 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.820941 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.821056 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.826284 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.826291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.826297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.826343 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.830502 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.831265 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.831271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.831277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.831450 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.832357 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.832363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.832367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.832783 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.832928 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.834079 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.834319 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":925,"threads_started":1}
I20260925 10:26:04.835114 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.808593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:51399 (local address 127.12.248.125:41103)
0925 10:26:04.809908 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.809913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.810673 (+ 760us) server_negotiation.cc:407] Connection header received
0925 10:26:04.813070 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.813076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.813166 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.813302 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.815904 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.816976 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.820637 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.821270 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.821369 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.827714 (+ 6345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.827723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.827729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.827794 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.831532 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.832472 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.832477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.832480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.832620 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.833076 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.833080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.833083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.833495 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.833660 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.834370 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.834630 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":1111,"threads_started":1}
I20260925 10:26:04.836191 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.809392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:53329)
0925 10:26:04.810503 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.810545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.812788 (+ 2243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.813765 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.813776 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.814658 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.815684 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.815709 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.818610 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.818616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.820168 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.820212 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.821615 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.821619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.821733 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.822238 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.822255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.826053 (+ 3798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.830687 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.830694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.830708 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.831133 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.831939 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.831943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.831947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.832250 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.833079 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.833089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.833643 (+ 554us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.833996 (+ 353us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.834605 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":83,"spinlock_wait_cycles":39552,"thread_start_us":1291,"threads_started":1}
I20260925 10:26:04.836236 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.809616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.12.248.66:51399)
0925 10:26:04.810500 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.810543 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.812792 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.813556 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.813567 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.814653 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.815705 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.815721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.817335 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.817343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.820491 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.820501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.822619 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.822629 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.822840 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.823444 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.823470 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.827473 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.831699 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.831704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.831708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.832302 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.832762 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.832767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.832770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.832933 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.833728 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.833733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.833993 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.835389 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.835683 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":127,"spinlock_wait_cycles":57856,"thread_start_us":418,"threads_started":1}
I20260925 10:26:04.836952 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.811677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:37215 (local address 127.12.248.124:45493)
0925 10:26:04.812231 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.812235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.812251 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:26:04.813158 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.813170 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.813234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.813349 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.815895 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.816891 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.819844 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.820376 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.820518 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.826696 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.826703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.826707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.826749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.830502 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.831925 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.831931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.831934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.832086 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.833153 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.833158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.833160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.833530 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.833680 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.836274 (+ 2594us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.836560 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
I20260925 10:26:04.837069 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.811434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.12.248.66:37215)
0925 10:26:04.812014 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:04.812035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:04.812788 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:04.813555 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:04.813563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:04.814658 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:04.815688 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.815709 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.817078 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.817083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.819663 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:04.819677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.820845 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.820852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:04.821147 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.822122 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:04.822160 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:04.826420 (+ 4260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:04.831311 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.831317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.831322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.831787 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.832235 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:04.832240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:04.832243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:04.833030 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:04.833743 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:04.833747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:04.834958 (+ 1211us) client_negotiation.cc:769] Sending connection context
0925 10:26:04.835133 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 10:26:04.835419 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":210,"spinlock_wait_cycles":16128,"thread_start_us":305,"threads_started":1}
I20260925 10:26:04.838261 20575 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:04.838301 20576 heartbeater.cc:347] Connected to a master server at 127.12.248.125:41103
I20260925 10:26:04.838655 20575 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.838654 20576 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.838802 20574 heartbeater.cc:347] Connected to a master server at 127.12.248.124:45493
I20260925 10:26:04.838955 20574 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:04.839838 20574 heartbeater.cc:511] Master 127.12.248.124:45493 requested a full tablet report, sending...
I20260925 10:26:04.840008 20575 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
I20260925 10:26:04.840929 20576 heartbeater.cc:511] Master 127.12.248.125:41103 requested a full tablet report, sending...
I20260925 10:26:04.841779 19462 ts_manager.cc:194] Re-registered known tserver with Master: 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:04.842406 20061 ts_manager.cc:194] Registered new tserver with Master: 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:04.842948 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:53329
I20260925 10:26:04.843109 20250 ts_manager.cc:194] Registered new tserver with Master: 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:04.844668 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:04.817407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.12.248.67:41331)
0925 10:26:04.819785 (+ 2378us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:04.819795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:04.819825 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:26:04.819886 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:04.819894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:04.820144 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:04.821248 (+ 1104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:04.822527 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.824052 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.825721 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:04.826410 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:04.826550 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:04.833379 (+ 6829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:04.833397 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:04.833417 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:04.833488 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:04.839654 (+ 6166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.840912 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.840929 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.840933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.841116 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:04.841697 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:04.841702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:04.841705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:04.842185 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:04.842564 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:04.843989 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0925 10:26:04.844270 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2194,"thread_start_us":2011,"threads_started":1}
I20260925 10:26:04.846051 19914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:04.931742 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19791
I20260925 10:26:04.948704 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:41331
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:38783
--webserver_port=41371
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:26:05.233045 20430 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
W20260925 10:26:05.289534 20587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:05.289841 20587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:05.290035 20587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:05.297675 20587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:05.297792 20587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:05.297986 20587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:05.307871 20587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:41331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=41371
--tserver_master_addrs=127.12.248.126:38783,127.12.248.125:41103,127.12.248.124:45493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:05.309333 20587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:05.310880 20587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:05.325182 20593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:05.325176 20595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:05.326066 20592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:05.333308 20587 server_base.cc:1034] running on GCE node
I20260925 10:26:05.334014 20587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:05.335114 20587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:05.336360 20587 hybrid_clock.cc:648] HybridClock initialized: now 1790331965336306 us; error 54 us; skew 500 ppm
I20260925 10:26:05.336691 20587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:05 dist-test-slave-f9rl krb5kdc[19417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331965, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:05.340142 20587 init.cc:377] Logged in from keytab as oryx/127.12.248.67@KRBTEST.COM (short username oryx)
I20260925 10:26:05.341989 20587 webserver.cc:460] Webserver started at http://127.12.248.67:41371/ using document root <none> and password file <none>
I20260925 10:26:05.342769 20587 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:05.342888 20587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:05.348908 20587 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 10:26:05.351708 20602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:05.353211 20587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:05.353413 20587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "91ee94d52ad541fcbaad403a1d627840"
format_stamp: "Formatted at 2026-09-25 10:26:01 on dist-test-slave-f9rl"
server_key: "1596e33475293a62fb5c5e137a3a7aec"
server_key_iv: "b63bb7c5d5da09ae8e8625e6fd3fa08c"
server_key_version: "encryptionkey@0"
I20260925 10:26:05.354460 20587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:05.367666 20587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:05.372084 20587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:05.372416 20587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:05.373384 20587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:05.374758 20587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:05.374845 20587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:05.374914 20587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:05.374969 20587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:05.409255 20587 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:41331
I20260925 10:26:05.409343 20714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:41331 every 8 connection(s)
I20260925 10:26:05.410816 20587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:05.419258 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20587
Sep 25 10:26:05 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331965, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.125@KRBTEST.COM
Sep 25 10:26:05 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331965, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
Sep 25 10:26:05 dist-test-slave-f9rl krb5kdc[19417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331965, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.124@KRBTEST.COM
I20260925 10:26:05.438673 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.413786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:34777 (local address 127.12.248.125:41103)
0925 10:26:05.414255 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.414261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.415659 (+ 1398us) server_negotiation.cc:407] Connection header received
0925 10:26:05.417960 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.417966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.418051 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.418176 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.420943 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.422208 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.425241 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.425964 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426066 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.431226 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.431231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.431234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.431280 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.434920 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.435875 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.435881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.435888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.436078 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.436603 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.436608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.436612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.436899 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.437051 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.437975 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.438222 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260925 10:26:05.440908 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.416119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38853 (local address 127.12.248.126:38783)
0925 10:26:05.416778 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.416782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.416799 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:05.418069 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.418075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.418151 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.418243 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.421996 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.423069 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.425247 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.425964 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426079 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.431795 (+ 5716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.431802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.431808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.431856 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.435784 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.438499 (+ 2715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.438505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.438508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.438662 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.439116 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.439121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.439124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.439477 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.439635 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.440296 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.440541 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":448,"threads_started":1}
I20260925 10:26:05.441586 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.414805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.12.248.67:34777)
0925 10:26:05.415555 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.415573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.417694 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.418439 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.418449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.419568 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:05.420665 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.420682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.422409 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.422414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.425083 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.425095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.426384 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.426781 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.427958 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:05.427986 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:05.431006 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:05.435094 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.435106 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.435125 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.435731 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.436261 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.436277 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.436280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.436485 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.437038 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:05.437047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:05.437388 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.438974 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.439553 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":344,"spinlock_wait_cycles":56448,"thread_start_us":408,"threads_started":1}
I20260925 10:26:05.441845 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.415970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:38853)
0925 10:26:05.416554 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.416570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.417694 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.418440 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.418450 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.419568 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:05.421821 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.421835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.423236 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.423243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.425081 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.425095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.426384 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.426770 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.427648 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:05.427683 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:05.431563 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:05.437481 (+ 5918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.437489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.437496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.438378 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.438792 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.438800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.438806 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.439000 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.439612 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:05.439620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:05.439958 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.440182 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.440556 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":254,"spinlock_wait_cycles":4736,"thread_start_us":310,"threads_started":1}
I20260925 10:26:05.443372 20717 heartbeater.cc:347] Connected to a master server at 127.12.248.125:41103
I20260925 10:26:05.443688 20717 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:05.443876 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.413832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:56107 (local address 127.12.248.124:45493)
0925 10:26:05.414628 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.414635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.415566 (+ 931us) server_negotiation.cc:407] Connection header received
0925 10:26:05.417981 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.417986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.418073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.418152 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.420884 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.422442 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.425359 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.426290 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426438 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.432785 (+ 6347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.432794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.432802 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.432855 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.437250 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.439667 (+ 2417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.439674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.439677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.439855 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.442121 (+ 2266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.442128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.442133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.442486 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.442608 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.443184 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.443435 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":592,"threads_started":1}
I20260925 10:26:05.444975 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.414446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.12.248.67:56107)
0925 10:26:05.415386 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.415445 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.417694 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.418440 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.418450 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.419569 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:05.420653 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.420679 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.422691 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.422699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.425145 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.425164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.426933 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.426945 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.427214 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.427916 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:05.427966 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:05.432511 (+ 4545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:05.437536 (+ 5025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.437540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.437544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.438318 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.440085 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:05.440091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:05.440094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:05.441534 (+ 1440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:05.442703 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:05.442710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:05.442901 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.444050 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.444349 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":266,"spinlock_wait_cycles":55168,"thread_start_us":373,"threads_started":1}
I20260925 10:26:05.445232 20716 heartbeater.cc:347] Connected to a master server at 127.12.248.126:38783
I20260925 10:26:05.445225 20717 heartbeater.cc:511] Master 127.12.248.125:41103 requested a full tablet report, sending...
I20260925 10:26:05.445406 20716 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:05.445976 20716 heartbeater.cc:511] Master 127.12.248.126:38783 requested a full tablet report, sending...
I20260925 10:26:05.446707 20715 heartbeater.cc:347] Connected to a master server at 127.12.248.124:45493
I20260925 10:26:05.446945 20715 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:05.447539 20715 heartbeater.cc:511] Master 127.12.248.124:45493 requested a full tablet report, sending...
I20260925 10:26:05.447557 20061 ts_manager.cc:194] Registered new tserver with Master: 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:05.448470 19462 ts_manager.cc:194] Re-registered known tserver with Master: 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:05.450165 19462 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:38853
I20260925 10:26:05.451444 20250 ts_manager.cc:194] Registered new tserver with Master: 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:05.471522 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.452354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.12.248.125:41103)
0925 10:26:05.453528 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.453533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.453783 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:26:05.454008 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.454013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.454104 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.454168 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.457557 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.458670 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.460629 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.461181 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.461337 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.464400 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.464407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.464412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.464454 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.468080 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.468999 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.469156 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.469616 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.469988 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.470283 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.470876 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.471088 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018}
I20260925 10:26:05.471805 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.453669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.12.248.124:45493)
0925 10:26:05.454328 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.454337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.455251 (+ 914us) server_negotiation.cc:407] Connection header received
0925 10:26:05.455452 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.455458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.455523 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.455624 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.457820 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.458890 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.460533 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.461185 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.461312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.465125 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.465133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.465142 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.465173 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.468168 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.469070 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.469238 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.469697 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.470024 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.470299 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.470979 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.471284 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260925 10:26:05.475005 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.452032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.12.248.126:38783)
0925 10:26:05.452197 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.452203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.453403 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 10:26:05.453721 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.453728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.453839 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.453913 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:05.456860 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.457975 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.460104 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.460742 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.460868 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.465042 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:05.465052 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:05.465062 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:05.465104 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:05.468446 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.469307 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.469469 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:05.469852 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:05.469857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:05.469859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:05.470257 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:05.470513 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.474267 (+ 3754us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.474495 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:26:05.478685 19462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:05.485713 19462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:05.527227 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.509546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.12.248.67:41331)
0925 10:26:05.514198 (+ 4652us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.514210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.514258 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:26:05.514396 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.514409 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.514792 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.515964 (+ 1172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.516413 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.518943 (+ 2530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.523990 (+ 5047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.525919 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.526104 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.526351 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.526648 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.526826 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4401,"thread_start_us":4146,"threads_started":1}
I20260925 10:26:05.527308 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.508854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:41331 (local address 127.0.0.1:39908)
0925 10:26:05.510979 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.511007 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.511176 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.515094 (+ 3918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.515103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.515158 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.515663 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.515680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.520410 (+ 4730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.520420 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.523751 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.523767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.526206 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.526214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.526418 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.526450 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.526591 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.526749 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":12220,"threads_started":1}
I20260925 10:26:05.531076 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.508096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.12.248.65:39655)
0925 10:26:05.520360 (+ 12264us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.520369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.520403 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:26:05.520540 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.520550 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.520983 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.521519 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.522549 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.524172 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.526769 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.529835 (+ 3066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.529999 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.530232 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.530542 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.530741 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12076,"thread_start_us":11946,"threads_started":1}
I20260925 10:26:05.531240 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.507933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:39655 (local address 127.0.0.1:44538)
0925 10:26:05.508857 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.508887 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.509062 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.521284 (+ 12222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.521289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.521323 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.522302 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.522317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.524320 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.524327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.526562 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.526578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.530063 (+ 3485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.530070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.530350 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.530376 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.530495 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.530658 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":82,"spinlock_wait_cycles":164736,"thread_start_us":536,"threads_started":1}
I20260925 10:26:05.531829 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.510472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.12.248.66:44403)
0925 10:26:05.519853 (+ 9381us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.519867 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.521679 (+ 1812us) server_negotiation.cc:407] Connection header received
0925 10:26:05.522015 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.522027 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.522290 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.522503 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.523537 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.525333 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.528230 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.529936 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.530128 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.530447 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.531338 (+ 891us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.531535 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9122,"thread_start_us":9177,"threads_started":1}
I20260925 10:26:05.531869 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.510188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:44403 (local address 127.0.0.1:57200)
0925 10:26:05.521482 (+ 11294us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.521504 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.521786 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.522741 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.522749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.522778 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.523240 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.523251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.525507 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.525513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.528027 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.528043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.530648 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.530654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.530855 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.530892 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.531011 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.531181 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11104,"thread_start_us":463,"threads_started":1}
I20260925 10:26:05.532673 20650 tablet_service.cc:1467] Processing CreateTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e (DEFAULT_TABLE table=test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:05.534691 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d77df3c0f64d24bb39401f4d4aae3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:05.535449 20364 tablet_service.cc:1467] Processing CreateTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e (DEFAULT_TABLE table=test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:05.536983 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d77df3c0f64d24bb39401f4d4aae3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:05.536854 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e (DEFAULT_TABLE table=test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:05.537986 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d77df3c0f64d24bb39401f4d4aae3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:05.552496 20746 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Bootstrap starting.
I20260925 10:26:05.554652 20748 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Bootstrap starting.
I20260925 10:26:05.556527 20747 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Bootstrap starting.
I20260925 10:26:05.560139 20748 tablet_bootstrap.cc:654] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:05.560333 20747 tablet_bootstrap.cc:654] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:05.560316 20746 tablet_bootstrap.cc:654] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:05.561400 20747 log.cc:824] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:05.561784 20746 log.cc:824] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:05.561775 20748 log.cc:824] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:05.564553 20747 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: No bootstrap required, opened a new log
I20260925 10:26:05.564908 20747 ts_tablet_manager.cc:1397] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 10:26:05.571906 20748 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: No bootstrap required, opened a new log
I20260925 10:26:05.572253 20748 ts_tablet_manager.cc:1397] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260925 10:26:05.572549 20746 tablet_bootstrap.cc:492] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: No bootstrap required, opened a new log
I20260925 10:26:05.572824 20746 ts_tablet_manager.cc:1397] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.007s
I20260925 10:26:05.576668 20747 raft_consensus.cc:348] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.577172 20747 raft_consensus.cc:374] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:05.577286 20747 raft_consensus.cc:729] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ee94d52ad541fcbaad403a1d627840, State: Initialized, Role: FOLLOWER
I20260925 10:26:05.578191 20747 consensus_queue.cc:260] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.582847 20748 raft_consensus.cc:348] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.583578 20748 raft_consensus.cc:374] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:05.583706 20748 raft_consensus.cc:729] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69421f70a24d423387c1b7f7d8db5e42, State: Initialized, Role: FOLLOWER
I20260925 10:26:05.584223 20747 ts_tablet_manager.cc:1428] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
I20260925 10:26:05.584542 20716 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
I20260925 10:26:05.584663 20748 consensus_queue.cc:260] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.585942 20746 raft_consensus.cc:348] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.587050 20746 raft_consensus.cc:374] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:05.587270 20746 raft_consensus.cc:729] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee6e4ff1fc44b49dcb69dcac5d0c19, State: Initialized, Role: FOLLOWER
I20260925 10:26:05.587935 20746 consensus_queue.cc:260] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.594592 20746 ts_tablet_manager.cc:1428] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Time spent starting tablet: real 0.022s user 0.011s sys 0.011s
I20260925 10:26:05.595112 20575 heartbeater.cc:503] Master 127.12.248.126:38783 was elected leader, sending a full tablet report...
I20260925 10:26:05.595369 20748 ts_tablet_manager.cc:1428] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Time spent starting tablet: real 0.023s user 0.016s sys 0.000s
W20260925 10:26:05.664712 20718 tablet.cc:2369] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:26:05.685856 20432 tablet.cc:2369] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:26:05.809110 20577 tablet.cc:2369] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:05.860040 20754 raft_consensus.cc:482] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:05.860355 20754 raft_consensus.cc:504] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.862166 20754 leader_election.cc:288] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655), 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:05.870241 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.862673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:56539 (local address 127.12.248.65:39655)
0925 10:26:05.862904 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.862909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.864098 (+ 1189us) server_negotiation.cc:407] Connection header received
0925 10:26:05.864438 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.864446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.864513 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.864619 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.865301 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.866148 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.868180 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.868979 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.869211 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.869316 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.869832 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.869963 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:26:05.870281 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.863942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:50831 (local address 127.12.248.67:41331)
0925 10:26:05.864148 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.864154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.864646 (+ 492us) server_negotiation.cc:407] Connection header received
0925 10:26:05.864824 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.864828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.864886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.864983 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.865630 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.866382 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.868269 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.869046 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.869222 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.869336 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.869857 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.870008 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:26:05.870492 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.863655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:41331 (local address 127.12.248.66:50831)
0925 10:26:05.864527 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.864541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.864692 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.864999 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.865003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.865025 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.865415 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.865423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.866517 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.866522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.868118 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.868130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.869409 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.869414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.869644 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.869697 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.869802 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.870020 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":135,"thread_start_us":469,"threads_started":1}
I20260925 10:26:05.870865 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.862666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:39655 (local address 127.12.248.66:56539)
0925 10:26:05.863956 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.863974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.864260 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.864675 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.864680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.864709 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.865140 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.865156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.866283 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.866288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.868017 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.868030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.869384 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.869390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.869665 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.869697 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.869805 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.869988 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":48,"thread_start_us":621,"threads_started":1}
I20260925 10:26:05.870779 20753 raft_consensus.cc:482] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:05.871083 20753 raft_consensus.cc:504] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.871714 20670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ee94d52ad541fcbaad403a1d627840" is_pre_election: true
I20260925 10:26:05.871938 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69421f70a24d423387c1b7f7d8db5e42" is_pre_election: true
I20260925 10:26:05.872453 20670 raft_consensus.cc:2454] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82ee6e4ff1fc44b49dcb69dcac5d0c19 in term 0.
I20260925 10:26:05.872428 20384 raft_consensus.cc:2454] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82ee6e4ff1fc44b49dcb69dcac5d0c19 in term 0.
I20260925 10:26:05.873220 20465 leader_election.cc:302] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ee6e4ff1fc44b49dcb69dcac5d0c19, 91ee94d52ad541fcbaad403a1d627840; no voters:
I20260925 10:26:05.873670 20754 raft_consensus.cc:2781] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:05.873823 20754 raft_consensus.cc:482] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:05.873937 20754 raft_consensus.cc:3037] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:05.874269 20753 leader_election.cc:288] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331), 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:05.877764 20754 raft_consensus.cc:504] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.878480 20754 leader_election.cc:288] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [CANDIDATE]: Term 1 election: Requested vote from peers 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655), 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331)
I20260925 10:26:05.878999 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69421f70a24d423387c1b7f7d8db5e42"
I20260925 10:26:05.879112 20670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ee94d52ad541fcbaad403a1d627840"
I20260925 10:26:05.879338 20384 raft_consensus.cc:3037] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:05.879431 20670 raft_consensus.cc:3037] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:05.883147 20670 raft_consensus.cc:2454] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82ee6e4ff1fc44b49dcb69dcac5d0c19 in term 1.
I20260925 10:26:05.883390 20384 raft_consensus.cc:2454] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82ee6e4ff1fc44b49dcb69dcac5d0c19 in term 1.
I20260925 10:26:05.883648 20465 leader_election.cc:302] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82ee6e4ff1fc44b49dcb69dcac5d0c19, 91ee94d52ad541fcbaad403a1d627840; no voters:
I20260925 10:26:05.884014 20754 raft_consensus.cc:2781] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:05.885334 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.874836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:47751 (local address 127.12.248.66:44403)
0925 10:26:05.875084 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.875092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.876037 (+ 945us) server_negotiation.cc:407] Connection header received
0925 10:26:05.876112 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.876117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.876192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.876289 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.876860 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.878038 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.882545 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.883961 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.884100 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.884176 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.884902 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.885039 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:26:05.885695 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.874438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:44403 (local address 127.12.248.65:47751)
0925 10:26:05.875675 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.875693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.875847 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.876324 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.876329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.876352 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.876684 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.876693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.878207 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.878213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.882340 (+ 4127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.882358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.884391 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.884400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.884670 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.884724 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.884857 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.885059 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":1271,"spinlock_wait_cycles":13696,"thread_start_us":953,"threads_started":1}
I20260925 10:26:05.886781 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.875935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49059 (local address 127.12.248.67:41331)
0925 10:26:05.876104 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.876110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.876127 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:05.876177 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.876180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.876229 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.876333 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:05.877056 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.878145 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.882920 (+ 4775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.884713 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.884864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.884934 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.886168 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.886417 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:26:05.887006 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.874424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:41331 (local address 127.12.248.65:49059)
0925 10:26:05.875454 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:05.875476 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:05.875610 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:05.876451 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:05.876456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:05.876502 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:05.876908 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.876922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.878289 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.878295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.882685 (+ 4390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:05.882700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.885309 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.885318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:05.885787 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.886024 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 10:26:05.886126 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:26:05.886289 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":6,"spinlock_wait_cycles":896,"thread_start_us":655,"threads_started":1}
I20260925 10:26:05.886943 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "69421f70a24d423387c1b7f7d8db5e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" is_pre_election: true
I20260925 10:26:05.887749 20670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d77df3c0f64d24bb39401f4d4aae3e" candidate_uuid: "69421f70a24d423387c1b7f7d8db5e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ee94d52ad541fcbaad403a1d627840" is_pre_election: true
I20260925 10:26:05.888126 20670 raft_consensus.cc:2379] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69421f70a24d423387c1b7f7d8db5e42 in current term 1: Already voted for candidate 82ee6e4ff1fc44b49dcb69dcac5d0c19 in this term.
I20260925 10:26:05.888085 20754 raft_consensus.cc:686] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 LEADER]: Becoming Leader. State: Replica: 82ee6e4ff1fc44b49dcb69dcac5d0c19, State: Running, Role: LEADER
I20260925 10:26:05.888800 20754 consensus_queue.cc:237] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } }
I20260925 10:26:05.893497 20319 leader_election.cc:302] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69421f70a24d423387c1b7f7d8db5e42; no voters: 82ee6e4ff1fc44b49dcb69dcac5d0c19, 91ee94d52ad541fcbaad403a1d627840
I20260925 10:26:05.894217 20753 raft_consensus.cc:2726] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:26:05.895165 19462 catalog_manager.cc:5495] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } health_report { overall_health: HEALTHY } } }
I20260925 10:26:05.974821 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:05.968958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.12.248.66:44403)
0925 10:26:05.969148 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:05.969154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:05.969174 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:26:05.969386 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:05.969390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:05.969527 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:05.969680 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:05.970035 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.970933 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.972382 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:05.972807 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:05.972942 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:05.973486 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:05.974147 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:05.974322 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:05.974388 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:26:05.974474 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:26:05.982673 20670 raft_consensus.cc:1261] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Refusing update from remote peer 82ee6e4ff1fc44b49dcb69dcac5d0c19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:05.982832 20384 raft_consensus.cc:1261] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Refusing update from remote peer 82ee6e4ff1fc44b49dcb69dcac5d0c19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:05.983667 20759 consensus_queue.cc:1035] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:05.984066 20754 consensus_queue.cc:1035] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:06.017592 19462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:06.017895 19462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:06.025108 19462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110d185250964711b03a7dc16bceaf87: Sending DeleteTablet for 3 replicas of tablet 33d77df3c0f64d24bb39401f4d4aae3e
I20260925 10:26:06.026324 20364 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:06 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:44538
I20260925 10:26:06.026553 20650 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:06 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:39908
I20260925 10:26:06.027935 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet 33d77df3c0f64d24bb39401f4d4aae3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:06 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:57200
I20260925 10:26:06.029088 20774 tablet_replica.cc:330] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: stopping tablet replica
I20260925 10:26:06.029871 20774 raft_consensus.cc:2229] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:06.028806 20772 tablet_replica.cc:330] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: stopping tablet replica
I20260925 10:26:06.030453 20772 raft_consensus.cc:2229] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:06.030897 20774 raft_consensus.cc:2258] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:06.031042 20772 raft_consensus.cc:2258] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:06.033267 20774 ts_tablet_manager.cc:1905] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:06.033933 20772 ts_tablet_manager.cc:1905] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:06.043040 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19430
I20260925 10:26:06.043483 20774 ts_tablet_manager.cc:1918] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:06.043826 20774 log.cc:1196] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/33d77df3c0f64d24bb39401f4d4aae3e
I20260925 10:26:06.044517 19492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:06.044848 20774 ts_tablet_manager.cc:1939] T 33d77df3c0f64d24bb39401f4d4aae3e P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting consensus metadata
I20260925 10:26:06.053164 20773 tablet_replica.cc:330] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: stopping tablet replica
I20260925 10:26:06.053865 20773 raft_consensus.cc:2229] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:06.054464 20773 raft_consensus.cc:2258] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:06.059327 20773 ts_tablet_manager.cc:1905] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:06.064497 20772 ts_tablet_manager.cc:1918] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:06.064860 20772 log.cc:1196] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/33d77df3c0f64d24bb39401f4d4aae3e
I20260925 10:26:06.065733 20772 ts_tablet_manager.cc:1939] T 33d77df3c0f64d24bb39401f4d4aae3e P 69421f70a24d423387c1b7f7d8db5e42: Deleting consensus metadata
I20260925 10:26:06.070869 20773 ts_tablet_manager.cc:1918] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:06.071053 20773 log.cc:1196] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/33d77df3c0f64d24bb39401f4d4aae3e
I20260925 10:26:06.071570 20773 ts_tablet_manager.cc:1939] T 33d77df3c0f64d24bb39401f4d4aae3e P 91ee94d52ad541fcbaad403a1d627840: Deleting consensus metadata
I20260925 10:26:06.084112 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:06.029697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.12.248.125:41103)
0925 10:26:06.056197 (+ 26500us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:06.056205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:06.056233 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:26:06.056310 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:06.056316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:06.056430 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:06.056597 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:06.065349 (+ 8752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.067338 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.069628 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.070654 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.075579 (+ 4925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:06.075933 (+ 354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:06.075941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:06.075946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:06.075983 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:06.080077 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.081237 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.081244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.081248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.081546 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.082076 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.082082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.082086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.082535 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:06.082886 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:06.083248 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 10:26:06.083594 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26342,"thread_start_us":26202,"threads_started":1}
I20260925 10:26:06.104607 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:06.038429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.12.248.124:45493)
0925 10:26:06.049132 (+ 10703us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:06.049142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:06.084053 (+ 34911us) server_negotiation.cc:407] Connection header received
0925 10:26:06.084203 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:06.084214 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:06.084317 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:06.084416 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:06.087445 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.088924 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.091287 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.092379 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.092509 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:06.096350 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:06.096358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:06.096364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:06.096412 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:06.100796 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.101885 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.101891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.101896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.102085 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.102588 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.102594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.102599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.103008 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:06.103295 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:06.103713 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 10:26:06.103959 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10511,"thread_start_us":10324,"threads_started":1}
I20260925 10:26:06.206724 19448 catalog_manager.cc:4841] TS 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403): tablet 33d77df3c0f64d24bb39401f4d4aae3e (table test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]) successfully deleted
I20260925 10:26:06.206900 19449 catalog_manager.cc:4841] TS 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331): tablet 33d77df3c0f64d24bb39401f4d4aae3e (table test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]) successfully deleted
I20260925 10:26:06.207213 19449 catalog_manager.cc:4841] TS 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655): tablet 33d77df3c0f64d24bb39401f4d4aae3e (table test-table [id=4e53a2bb7f724e68ac3aac2d09bd77d9]) successfully deleted
W20260925 10:26:06.207042 19448 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.248.66:44403, user_credentials={real_user=slave}} blocked reactor thread for 159577us
I20260925 10:26:06.207494 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19430
I20260925 10:26:06.252135 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:06.236236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.12.248.124:45493)
0925 10:26:06.236439 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:06.236447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:06.236465 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:26:06.236533 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:06.236540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:06.236633 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:06.236769 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:06.238301 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.239275 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.241173 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.241739 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.241872 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:06.244744 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:06.244751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:06.244755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:06.244794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:06.249228 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.250154 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.250160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.250163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.250295 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.250687 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.250693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.250696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.250972 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:06.251242 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:06.251553 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:26:06.251762 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:26:06.255162 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:06.234503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.12.248.125:41103)
0925 10:26:06.234740 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:06.234748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:06.235361 (+ 613us) server_negotiation.cc:407] Connection header received
0925 10:26:06.235742 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:06.235749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:06.235843 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:06.235922 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:06.238741 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.239772 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.242078 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:06.242673 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:06.242830 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:06.247859 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:06.247865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:06.247870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:06.247914 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:06.251461 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.252598 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.252607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.252611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.252770 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:06.253496 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:06.253502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:06.253506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:06.253869 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:06.254155 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:06.254524 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 10:26:06.254793 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:26:07.012418 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.011471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:45237)
0925 10:26:07.012029 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.012185 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":319,"threads_started":1}
W20260925 10:26:07.014299 20716 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:38783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.014360 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.012508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:45331)
0925 10:26:07.013820 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.014006 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1159,"threads_started":1}
W20260925 10:26:07.015913 20575 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:38783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.016142 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.013250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:42635)
0925 10:26:07.015772 (+ 2522us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.015918 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2383,"thread_start_us":2280,"threads_started":1}
W20260925 10:26:07.016558 20430 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:38783 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.526015 20789 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110d185250964711b03a7dc16bceaf87)
I20260925 10:26:07.526170 20790 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110d185250964711b03a7dc16bceaf87)
I20260925 10:26:07.526400 20789 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:07.526429 20790 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:07.527420 20790 leader_election.cc:288] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103)
I20260925 10:26:07.534374 20789 leader_election.cc:288] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), f72d5c990ecf403db8800cee4d2e735a (127.12.248.124:45493)
I20260925 10:26:07.541610 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.528226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49380)
0925 10:26:07.541098 (+ 12872us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.541276 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12742,"threads_started":1}
W20260925 10:26:07.542484 20046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.544643 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.527662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49378)
0925 10:26:07.543601 (+ 15939us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.543951 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15771,"thread_start_us":11858,"threads_started":1}
W20260925 10:26:07.545120 20235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.548466 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.546450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49396)
0925 10:26:07.547934 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.548139 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":1126,"threads_started":1}
W20260925 10:26:07.549908 20046 leader_election.cc:334] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.550619 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.550222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49406)
0925 10:26:07.550381 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.550482 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 10:26:07.550796 20235 leader_election.cc:334] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.552484 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.527824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.12.248.125:41103)
0925 10:26:07.540333 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:07.540341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:07.540370 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:26:07.540462 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:07.540470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:07.540580 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:07.540752 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:07.541603 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.542831 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.549525 (+ 6694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.551402 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.551586 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:07.551671 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:07.552017 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:26:07.552143 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":412,"server-negotiator.queue_time_us":12330,"thread_start_us":12128,"threads_started":1}
I20260925 10:26:07.552572 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.527646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41103 (local address 127.0.0.1:33444)
0925 10:26:07.539882 (+ 12236us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.539913 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:07.540100 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:07.540989 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:07.540995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:07.541025 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:07.541340 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:07.541355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.543048 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.543056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:07.549276 (+ 6220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:07.549299 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.551601 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.551608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:07.551788 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:07.551864 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 10:26:07.551974 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:26:07.552118 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12058,"mutex_wait_us":42,"thread_start_us":16370,"threads_started":1}
I20260925 10:26:07.553411 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.542510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.12.248.124:45493)
0925 10:26:07.546182 (+ 3672us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:07.546187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:07.546205 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:26:07.546256 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:07.546260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:07.546324 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:07.546420 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:07.547479 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.549263 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.551577 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.552432 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.552623 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:07.552692 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:07.552974 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:26:07.553068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3551,"thread_start_us":2673,"threads_started":1}
I20260925 10:26:07.553409 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.542222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45493 (local address 127.0.0.1:44598)
0925 10:26:07.542998 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.543017 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:07.543115 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:07.546803 (+ 3688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:07.546808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:07.546857 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:07.547308 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:07.547323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.549399 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.549405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:07.551405 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:07.551419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:07.552613 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:07.552619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:07.552796 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:07.552843 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:26:07.552943 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:26:07.553079 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":71}
I20260925 10:26:07.553370 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" is_pre_election: true
I20260925 10:26:07.553694 20071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f72d5c990ecf403db8800cee4d2e735a in term 1.
I20260925 10:26:07.554181 20236 leader_election.cc:302] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc4ee7b2a2974ec29d0a076103cfefe9, f72d5c990ecf403db8800cee4d2e735a; no voters: 110d185250964711b03a7dc16bceaf87
I20260925 10:26:07.554178 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f72d5c990ecf403db8800cee4d2e735a" is_pre_election: true
I20260925 10:26:07.554519 20790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:26:07.554664 20260 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4ee7b2a2974ec29d0a076103cfefe9 in term 1.
I20260925 10:26:07.554975 20790 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110d185250964711b03a7dc16bceaf87)
I20260925 10:26:07.555101 20790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:26:07.555217 20046 leader_election.cc:302] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc4ee7b2a2974ec29d0a076103cfefe9, f72d5c990ecf403db8800cee4d2e735a; no voters: 110d185250964711b03a7dc16bceaf87
I20260925 10:26:07.555562 20789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:26:07.555652 20789 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110d185250964711b03a7dc16bceaf87)
I20260925 10:26:07.555711 20789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:26:07.557314 20790 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:07.557996 20790 leader_election.cc:288] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 election: Requested vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103)
I20260925 10:26:07.557971 20789 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:07.558531 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.558136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49412)
0925 10:26:07.558307 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.558397 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:26:07.558699 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
I20260925 10:26:07.558868 20789 leader_election.cc:288] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 election: Requested vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), f72d5c990ecf403db8800cee4d2e735a (127.12.248.124:45493)
I20260925 10:26:07.559010 20071 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f72d5c990ecf403db8800cee4d2e735a in current term 2: Already voted for candidate bc4ee7b2a2974ec29d0a076103cfefe9 in this term.
I20260925 10:26:07.559721 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.559194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49422)
0925 10:26:07.559487 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.559551 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 10:26:07.559719 20260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f72d5c990ecf403db8800cee4d2e735a"
I20260925 10:26:07.559981 20260 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bc4ee7b2a2974ec29d0a076103cfefe9 in current term 2: Already voted for candidate f72d5c990ecf403db8800cee4d2e735a in this term.
I20260925 10:26:07.560276 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.559804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49424)
0925 10:26:07.559995 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.560090 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 10:26:07.560436 20235 leader_election.cc:334] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.560566 20235 leader_election.cc:302] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f72d5c990ecf403db8800cee4d2e735a; no voters: 110d185250964711b03a7dc16bceaf87, bc4ee7b2a2974ec29d0a076103cfefe9
I20260925 10:26:07.560789 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:07.560396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49438)
0925 10:26:07.560543 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:07.560624 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:26:07.560825 20790 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 10:26:07.561004 20046 leader_election.cc:334] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:07.561141 20046 leader_election.cc:302] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc4ee7b2a2974ec29d0a076103cfefe9; no voters: 110d185250964711b03a7dc16bceaf87, f72d5c990ecf403db8800cee4d2e735a
I20260925 10:26:07.561473 20789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 10:26:08.016168 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:08.015210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:33607)
0925 10:26:08.015788 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:08.015919 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":630,"threads_started":1}
I20260925 10:26:08.018148 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:08.016634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:48943)
0925 10:26:08.017252 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:08.017366 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":374,"threads_started":1}
I20260925 10:26:08.018504 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:08.017213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:37445)
0925 10:26:08.018174 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:08.018284 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":707,"threads_started":1}
I20260925 10:26:09.018400 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.017512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:58219)
0925 10:26:09.017983 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.018118 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260925 10:26:09.020457 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.019089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:41659)
0925 10:26:09.020168 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.020288 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":809,"threads_started":1}
I20260925 10:26:09.020540 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.019266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:52625)
0925 10:26:09.020302 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.020390 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":825,"threads_started":1}
I20260925 10:26:09.424685 20805 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:09.424932 20805 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:09.425859 20805 leader_election.cc:288] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103)
I20260925 10:26:09.426538 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" is_pre_election: true
I20260925 10:26:09.426790 20071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f72d5c990ecf403db8800cee4d2e735a in term 2.
I20260925 10:26:09.426934 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.426072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49732)
0925 10:26:09.426575 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.426699 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260925 10:26:09.427398 20236 leader_election.cc:302] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc4ee7b2a2974ec29d0a076103cfefe9, f72d5c990ecf403db8800cee4d2e735a; no voters:
I20260925 10:26:09.427683 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 10:26:09.427827 20805 raft_consensus.cc:482] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:09.427910 20805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:26:09.428824 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.428405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49740)
0925 10:26:09.428619 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.428702 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 10:26:09.429095 20235 leader_election.cc:334] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:09.430661 20805 raft_consensus.cc:504] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:09.431262 20805 leader_election.cc:288] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 election: Requested vote from peers 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783), bc4ee7b2a2974ec29d0a076103cfefe9 (127.12.248.125:41103)
I20260925 10:26:09.431789 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.431343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49748)
0925 10:26:09.431593 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.431669 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260925 10:26:09.431902 20071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f72d5c990ecf403db8800cee4d2e735a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9"
I20260925 10:26:09.432226 20071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:26:09.432749 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.432351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49760)
0925 10:26:09.432515 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.432617 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 10:26:09.432919 20235 leader_election.cc:334] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
I20260925 10:26:09.434646 20071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f72d5c990ecf403db8800cee4d2e735a in term 3.
I20260925 10:26:09.435081 20236 leader_election.cc:302] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc4ee7b2a2974ec29d0a076103cfefe9, f72d5c990ecf403db8800cee4d2e735a; no voters: 110d185250964711b03a7dc16bceaf87
I20260925 10:26:09.435398 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:26:09.435827 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [term 3 LEADER]: Becoming Leader. State: Replica: f72d5c990ecf403db8800cee4d2e735a, State: Running, Role: LEADER
I20260925 10:26:09.436182 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } }
I20260925 10:26:09.438220 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72d5c990ecf403db8800cee4d2e735a. Latest consensus state: current_term: 3 leader_uuid: "f72d5c990ecf403db8800cee4d2e735a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:09.438560 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:09.439240 20810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:09.442902 20810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:09.443707 20810 catalog_manager.cc:1259] Loaded cluster ID: 4d36fe76d3444ea7a64880ca0becff65
I20260925 10:26:09.443778 20810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:09.445047 20810 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:09.445948 20810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Loaded TSK: 0
I20260925 10:26:09.446889 20810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:26:09.475574 20250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:09.483346 20250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:09.488010 20071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [term 3 FOLLOWER]: Refusing update from remote peer f72d5c990ecf403db8800cee4d2e735a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 10:26:09.488178 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.487603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49770)
0925 10:26:09.487808 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.487894 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 10:26:09.488843 20808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 10:26:09.489293 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.488919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49774)
0925 10:26:09.489073 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.489172 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:26:09.489503 20235 consensus_peers.cc:487] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a -> Peer 110d185250964711b03a7dc16bceaf87 (127.12.248.126:38783): Couldn't send request to peer 110d185250964711b03a7dc16bceaf87. Status: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:26:09.492398 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72d5c990ecf403db8800cee4d2e735a. Latest consensus state: current_term: 3 leader_uuid: "f72d5c990ecf403db8800cee4d2e735a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:09.492671 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:09.493541 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f72d5c990ecf403db8800cee4d2e735a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:09.493844 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:09.494232 20813 mvcc.cc:205] Tried to move back new op lower bound from 7333199747015557120 to 7333199746813882368. Current Snapshot: MvccSnapshot[applied={T|T < 7333199747015557120}]
I20260925 10:26:09.494764 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f72d5c990ecf403db8800cee4d2e735a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "110d185250964711b03a7dc16bceaf87" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38783 } } peers { permanent_uuid: "bc4ee7b2a2974ec29d0a076103cfefe9" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41103 } attrs { promote: false } } peers { permanent_uuid: "f72d5c990ecf403db8800cee4d2e735a" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45493 } attrs { promote: false } } }
I20260925 10:26:09.494999 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4ee7b2a2974ec29d0a076103cfefe9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:26:09.520491 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.502370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.12.248.65:39655)
0925 10:26:09.509115 (+ 6745us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.509128 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.509162 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:26:09.509292 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.509302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.509448 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.511340 (+ 1892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:09.511612 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.513155 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.517289 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.519379 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.519536 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.519664 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.519980 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.520129 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6512,"thread_start_us":657,"threads_started":1}
I20260925 10:26:09.520491 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.502249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:39655 (local address 127.0.0.1:44548)
0925 10:26:09.502435 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.502458 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:09.502620 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:09.510523 (+ 7903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:09.510534 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:09.510579 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:09.511099 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.511119 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.515000 (+ 3881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.515007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.517015 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.517025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.519618 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.519625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.519803 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.519831 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:26:09.519952 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:26:09.520099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":27}
I20260925 10:26:09.522330 20364 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (DEFAULT_TABLE table=test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:09.522864 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc878ec78ca4ed2bbdcb833e7ded586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:09.523188 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.503030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.12.248.67:41331)
0925 10:26:09.515429 (+ 12399us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.515436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.515458 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:26:09.515531 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.515537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.515632 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.515777 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:09.516572 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.517853 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.520824 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.522086 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.522204 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.522324 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.522780 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.522915 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12262,"thread_start_us":12134,"threads_started":1}
I20260925 10:26:09.523314 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.502920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:41331 (local address 127.0.0.1:39914)
0925 10:26:09.514340 (+ 11420us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.514367 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:09.514502 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:09.515833 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:09.515840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:09.515870 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:09.516329 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.516341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.518333 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.518338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.520622 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.520641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.522430 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.522435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.522591 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.522625 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:26:09.522737 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:26:09.522867 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11260,"mutex_wait_us":35,"spinlock_wait_cycles":3200,"thread_start_us":10942,"threads_started":1}
I20260925 10:26:09.524034 20650 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (DEFAULT_TABLE table=test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:09.524227 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.502752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:44403 (local address 127.0.0.1:57218)
0925 10:26:09.514286 (+ 11534us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.514318 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:09.514495 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:09.515487 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:09.515492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:09.515518 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:09.517121 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.517127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.518582 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.518588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.520753 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.520766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.523124 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.523131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.523508 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.523563 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 10:26:09.523666 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:26:09.523799 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11332,"mutex_wait_us":4,"spinlock_wait_cycles":1280,"thread_start_us":11844,"threads_started":1}
I20260925 10:26:09.524394 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc878ec78ca4ed2bbdcb833e7ded586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:09.524989 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.503138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.12.248.66:44403)
0925 10:26:09.515214 (+ 12076us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.515218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.515234 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:26:09.515291 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.515295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.515352 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.515472 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:09.517293 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.518406 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.520911 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.522681 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.522823 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.522954 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.524445 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.524667 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11950,"thread_start_us":11804,"threads_started":1}
I20260925 10:26:09.525738 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (DEFAULT_TABLE table=test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:09.526221 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc878ec78ca4ed2bbdcb833e7ded586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:09.530016 20822 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Bootstrap starting.
I20260925 10:26:09.531896 20822 tablet_bootstrap.cc:654] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:09.534137 20822 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: No bootstrap required, opened a new log
I20260925 10:26:09.534344 20822 ts_tablet_manager.cc:1397] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:26:09.535509 20822 raft_consensus.cc:348] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.535908 20822 raft_consensus.cc:374] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:09.536037 20822 raft_consensus.cc:729] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ee94d52ad541fcbaad403a1d627840, State: Initialized, Role: FOLLOWER
I20260925 10:26:09.536391 20822 consensus_queue.cc:260] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.537035 20824 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Bootstrap starting.
I20260925 10:26:09.538717 20822 ts_tablet_manager.cc:1428] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260925 10:26:09.539294 20825 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Bootstrap starting.
I20260925 10:26:09.539410 20824 tablet_bootstrap.cc:654] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:09.542971 20825 tablet_bootstrap.cc:654] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:09.546851 20715 heartbeater.cc:503] Master 127.12.248.124:45493 was elected leader, sending a full tablet report...
I20260925 10:26:09.547519 20825 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: No bootstrap required, opened a new log
I20260925 10:26:09.547550 20824 tablet_bootstrap.cc:492] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: No bootstrap required, opened a new log
I20260925 10:26:09.547685 20824 ts_tablet_manager.cc:1397] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 10:26:09.547685 20825 ts_tablet_manager.cc:1397] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260925 10:26:09.548532 20824 raft_consensus.cc:348] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.548529 20825 raft_consensus.cc:348] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.548777 20824 raft_consensus.cc:374] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:09.548789 20825 raft_consensus.cc:374] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:09.548882 20824 raft_consensus.cc:729] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69421f70a24d423387c1b7f7d8db5e42, State: Initialized, Role: FOLLOWER
I20260925 10:26:09.548882 20825 raft_consensus.cc:729] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82ee6e4ff1fc44b49dcb69dcac5d0c19, State: Initialized, Role: FOLLOWER
I20260925 10:26:09.549137 20825 consensus_queue.cc:260] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.549499 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.539962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:49285)
0925 10:26:09.546680 (+ 6718us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.549142 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6546,"thread_start_us":6419,"threads_started":1}
I20260925 10:26:09.549787 20824 consensus_queue.cc:260] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.550081 20825 ts_tablet_manager.cc:1428] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:26:09.550977 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.550477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:36801)
0925 10:26:09.550701 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.550811 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 10:26:09.551988 20824 ts_tablet_manager.cc:1428] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 10:26:09.555006 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.550971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:37305)
0925 10:26:09.554541 (+ 3570us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.554699 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3444,"thread_start_us":3302,"threads_started":1}
I20260925 10:26:09.555110 20574 heartbeater.cc:503] Master 127.12.248.124:45493 was elected leader, sending a full tablet report...
I20260925 10:26:09.556949 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.554385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:59681)
0925 10:26:09.556495 (+ 2110us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.556649 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1953,"thread_start_us":2261,"threads_started":1}
I20260925 10:26:09.556957 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.556026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:41549)
0925 10:26:09.556708 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.556803 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576}
I20260925 10:26:09.557165 20429 heartbeater.cc:503] Master 127.12.248.124:45493 was elected leader, sending a full tablet report...
I20260925 10:26:09.721676 20826 raft_consensus.cc:482] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:09.721909 20826 raft_consensus.cc:504] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.723531 20826 leader_election.cc:288] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655), 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:09.730618 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.723853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:53837 (local address 127.12.248.65:39655)
0925 10:26:09.724075 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.724081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.724423 (+ 342us) server_negotiation.cc:407] Connection header received
0925 10:26:09.724677 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.724684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.724765 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.724854 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:09.725423 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.726387 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.728575 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.729675 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.729802 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.729854 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.730236 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.730366 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:26:09.730690 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.723757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:39655 (local address 127.12.248.67:53837)
0925 10:26:09.724249 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.724271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:09.724507 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:09.724895 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:09.724901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:09.724936 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:09.725265 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.725280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.726522 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.726529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.728422 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.728435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.729854 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.729862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.730049 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.730088 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:26:09.730184 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:26:09.730342 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":1}
I20260925 10:26:09.731159 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.723979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:51267 (local address 127.12.248.66:44403)
0925 10:26:09.724174 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.724180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.724742 (+ 562us) server_negotiation.cc:407] Connection header received
0925 10:26:09.724905 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.724911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.724989 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.725197 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:09.725867 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.726763 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.728844 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.730061 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.730239 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.730378 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.730706 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.730852 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:26:09.731235 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.723873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:44403 (local address 127.12.248.67:51267)
0925 10:26:09.724620 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.724636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:09.724764 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:09.725247 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:09.725253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:09.725280 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:09.725701 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.725712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.726887 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.726891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.728667 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:09.728681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.730285 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.730292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:09.730449 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.730482 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:26:09.730650 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:26:09.730805 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":28,"thread_start_us":664,"threads_started":1}
I20260925 10:26:09.731582 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc878ec78ca4ed2bbdcb833e7ded586" candidate_uuid: "91ee94d52ad541fcbaad403a1d627840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69421f70a24d423387c1b7f7d8db5e42" is_pre_election: true
I20260925 10:26:09.731869 20384 raft_consensus.cc:2454] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ee94d52ad541fcbaad403a1d627840 in term 0.
I20260925 10:26:09.732440 20606 leader_election.cc:302] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69421f70a24d423387c1b7f7d8db5e42, 91ee94d52ad541fcbaad403a1d627840; no voters:
I20260925 10:26:09.732667 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc878ec78ca4ed2bbdcb833e7ded586" candidate_uuid: "91ee94d52ad541fcbaad403a1d627840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" is_pre_election: true
I20260925 10:26:09.732932 20826 raft_consensus.cc:2781] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:09.732959 20529 raft_consensus.cc:2454] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ee94d52ad541fcbaad403a1d627840 in term 0.
I20260925 10:26:09.733067 20826 raft_consensus.cc:482] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:09.733178 20826 raft_consensus.cc:3037] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:09.735461 20826 raft_consensus.cc:504] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.736022 20826 leader_election.cc:288] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [CANDIDATE]: Term 1 election: Requested vote from peers 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655), 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403)
I20260925 10:26:09.736487 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc878ec78ca4ed2bbdcb833e7ded586" candidate_uuid: "91ee94d52ad541fcbaad403a1d627840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19"
I20260925 10:26:09.736569 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc878ec78ca4ed2bbdcb833e7ded586" candidate_uuid: "91ee94d52ad541fcbaad403a1d627840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69421f70a24d423387c1b7f7d8db5e42"
I20260925 10:26:09.736713 20529 raft_consensus.cc:3037] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:09.736824 20384 raft_consensus.cc:3037] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:09.738897 20529 raft_consensus.cc:2454] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ee94d52ad541fcbaad403a1d627840 in term 1.
I20260925 10:26:09.738914 20384 raft_consensus.cc:2454] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ee94d52ad541fcbaad403a1d627840 in term 1.
I20260925 10:26:09.739300 20606 leader_election.cc:302] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69421f70a24d423387c1b7f7d8db5e42, 91ee94d52ad541fcbaad403a1d627840; no voters:
I20260925 10:26:09.739566 20826 raft_consensus.cc:2781] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:09.740139 20826 raft_consensus.cc:686] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 1 LEADER]: Becoming Leader. State: Replica: 91ee94d52ad541fcbaad403a1d627840, State: Running, Role: LEADER
I20260925 10:26:09.740586 20826 consensus_queue.cc:237] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } }
I20260925 10:26:09.741205 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.740757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:43483)
0925 10:26:09.740996 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.741078 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:26:09.744532 20249 catalog_manager.cc:5495] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "91ee94d52ad541fcbaad403a1d627840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ee94d52ad541fcbaad403a1d627840" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 41331 } health_report { overall_health: HEALTHY } } }
I20260925 10:26:09.766999 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.760648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.12.248.67:41331)
0925 10:26:09.760907 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.760912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.760930 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:26:09.761245 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.761251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.761352 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.761618 (+ 266us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:09.761923 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.762820 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.764099 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.764576 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.764691 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.765120 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:09.766265 (+ 1145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:09.766402 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.766579 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.766658 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260925 10:26:09.771270 20384 raft_consensus.cc:1261] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Refusing update from remote peer 91ee94d52ad541fcbaad403a1d627840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:09.771521 20529 raft_consensus.cc:1261] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Refusing update from remote peer 91ee94d52ad541fcbaad403a1d627840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:09.772045 20835 consensus_queue.cc:1035] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69421f70a24d423387c1b7f7d8db5e42" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:09.772496 20826 consensus_queue.cc:1035] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82ee6e4ff1fc44b49dcb69dcac5d0c19" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 44403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:09.774770 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.774296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:60525)
0925 10:26:09.774502 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.774606 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 10:26:09.776170 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.775675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:35623)
0925 10:26:09.775871 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.775965 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 10:26:09.782243 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.781412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:35961)
0925 10:26:09.781930 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.782030 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410}
I20260925 10:26:09.782716 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.782247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.66:45833)
0925 10:26:09.782460 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.782543 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 10:26:09.784260 20838 mvcc.cc:205] Tried to move back new op lower bound from 7333199748175970304 to 7333199748061597696. Current Snapshot: MvccSnapshot[applied={T|T < 7333199748175970304 or (T in {7333199748175970304})}]
I20260925 10:26:09.784641 20840 mvcc.cc:205] Tried to move back new op lower bound from 7333199748175970304 to 7333199748061597696. Current Snapshot: MvccSnapshot[applied={T|T < 7333199748175970304}]
I20260925 10:26:09.791461 20839 mvcc.cc:205] Tried to move back new op lower bound from 7333199748175970304 to 7333199748061597696. Current Snapshot: MvccSnapshot[applied={T|T < 7333199748175970304}]
I20260925 10:26:09.793000 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.792407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.65:56679)
0925 10:26:09.792766 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.792871 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250}
I20260925 10:26:09.794098 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.790935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.12.248.67:47121)
0925 10:26:09.793792 (+ 2857us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:09.793912 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2742}
I20260925 10:26:09.800688 20250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:09.800948 20250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:09.809903 20250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f72d5c990ecf403db8800cee4d2e735a: Sending DeleteTablet for 3 replicas of tablet 3bc878ec78ca4ed2bbdcb833e7ded586
I20260925 10:26:09.811414 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:09 UTC) from {username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:57218
I20260925 10:26:09.811887 20650 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:09 UTC) from {username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:39914
I20260925 10:26:09.811875 20364 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc878ec78ca4ed2bbdcb833e7ded586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:09 UTC) from {username='oryx', principal='oryx/127.12.248.124@KRBTEST.COM'} at 127.0.0.1:44548
I20260925 10:26:09.814198 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20229
I20260925 10:26:09.843688 20850 tablet_replica.cc:330] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: stopping tablet replica
I20260925 10:26:09.844105 20850 raft_consensus.cc:2229] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:09.845186 20850 raft_consensus.cc:2258] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:09.845957 20849 tablet_replica.cc:330] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: stopping tablet replica
I20260925 10:26:09.846216 20849 raft_consensus.cc:2229] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:09.847112 20849 raft_consensus.cc:2258] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:09.847693 20851 tablet_replica.cc:330] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: stopping tablet replica
I20260925 10:26:09.847900 20851 raft_consensus.cc:2229] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:09.848215 20851 raft_consensus.cc:2258] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:09.849874 20851 ts_tablet_manager.cc:1905] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:09.855314 20849 ts_tablet_manager.cc:1905] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:09.858196 20850 ts_tablet_manager.cc:1905] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:09.858836 20851 ts_tablet_manager.cc:1918] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:09.859031 20851 log.cc:1196] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/3bc878ec78ca4ed2bbdcb833e7ded586
I20260925 10:26:09.859597 20851 ts_tablet_manager.cc:1939] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 69421f70a24d423387c1b7f7d8db5e42: Deleting consensus metadata
I20260925 10:26:09.860931 20237 catalog_manager.cc:4841] TS 69421f70a24d423387c1b7f7d8db5e42 (127.12.248.65:39655): tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (table test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]) successfully deleted
I20260925 10:26:09.865897 20849 ts_tablet_manager.cc:1918] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:09.866056 20849 log.cc:1196] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/3bc878ec78ca4ed2bbdcb833e7ded586
I20260925 10:26:09.866627 20849 ts_tablet_manager.cc:1939] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 82ee6e4ff1fc44b49dcb69dcac5d0c19: Deleting consensus metadata
I20260925 10:26:09.867933 20236 catalog_manager.cc:4841] TS 82ee6e4ff1fc44b49dcb69dcac5d0c19 (127.12.248.66:44403): tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (table test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]) successfully deleted
I20260925 10:26:09.869170 20850 ts_tablet_manager.cc:1918] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:09.869333 20850 log.cc:1196] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/3bc878ec78ca4ed2bbdcb833e7ded586
I20260925 10:26:09.869858 20850 ts_tablet_manager.cc:1939] T 3bc878ec78ca4ed2bbdcb833e7ded586 P 91ee94d52ad541fcbaad403a1d627840: Deleting consensus metadata
I20260925 10:26:09.871004 20237 catalog_manager.cc:4841] TS 91ee94d52ad541fcbaad403a1d627840 (127.12.248.67:41331): tablet 3bc878ec78ca4ed2bbdcb833e7ded586 (table test-table [id=711cbbfd5a914d1fbfd66f64c1513ad3]) successfully deleted
I20260925 10:26:09.873697 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:09.815213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.12.248.65:39655)
0925 10:26:09.815394 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:09.815400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:09.858059 (+ 42659us) server_negotiation.cc:407] Connection header received
0925 10:26:09.858359 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:09.858366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:09.858452 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:09.858567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:09.860730 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.861668 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.863242 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:09.863655 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:09.863805 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:09.867187 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:09.867209 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:09.867229 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:09.867269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:09.870710 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:09.871545 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:09.871555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:09.871559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:09.871703 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:09.872084 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:09.872092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:09.872095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:09.872452 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:09.872783 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:09.873083 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 10:26:09.873329 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:26:09.874341 20424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:09.991873 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20229
I20260925 10:26:10.009907 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20441
I20260925 10:26:10.024382 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.010918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.12.248.66:44403)
0925 10:26:10.011125 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:10.011133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:10.011153 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:26:10.011356 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:10.011362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:10.011439 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:10.011552 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:10.013007 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:10.013850 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:10.015114 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:10.015541 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:10.015699 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:10.018461 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:10.018482 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:10.018498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:10.018535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:10.021553 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:10.022227 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:10.022236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:10.022240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:10.022382 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:10.022763 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:10.022769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:10.022773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:10.023137 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:10.023461 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:10.023769 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:26:10.024013 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:26:10.025224 20569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:10.048638 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.047594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49788)
0925 10:26:10.048312 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:10.048417 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":458,"threads_started":1}
I20260925 10:26:10.050159 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.049784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49794)
0925 10:26:10.049942 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:10.050029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:26:10.156476 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20441
I20260925 10:26:10.179160 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20587
I20260925 10:26:10.196512 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.180063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.12.248.67:41331)
0925 10:26:10.180277 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:10.180285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:10.180378 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:26:10.180546 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:10.180552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:10.180667 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:10.180805 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:10.182536 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:10.183805 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:10.185142 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:10.185890 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:10.186031 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:10.188917 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:10.188936 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:10.188958 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:10.189002 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:10.193369 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:10.194158 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:10.194167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:10.194171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:10.194387 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:10.194757 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:10.194764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:10.194770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:10.195272 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:10.195623 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:10.195852 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:26:10.196087 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:26:10.197316 20710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:10.331513 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20587
I20260925 10:26:10.352644 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 19924
I20260925 10:26:10.353896 20091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:10.426167 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.425719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49798)
0925 10:26:10.425927 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:10.426020 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 10:26:10.427301 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:10.426949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38783 (local address 127.0.0.1:49804)
0925 10:26:10.427105 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:10.427182 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:26:10.474224 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 19924
I20260925 10:26:10.495265 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20024
I20260925 10:26:10.496312 20280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:10.618798 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20024
2026-09-25T10:26:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12053 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:26:10 dist-test-slave-f9rl krb5kdc[20862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:26:10 dist-test-slave-f9rl krb5kdc[20862](info): set up 2 sockets
Sep 25 10:26:10 dist-test-slave-f9rl krb5kdc[20862](info): commencing operation
krb5kdc: starting...
W20260925 10:26:11.772778 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:26:11 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab.
Entry for principal oryx/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab.
I20260925 10:26:11.997928 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37229
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab
--principal=oryx/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:12.327791 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:12.328091 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:12.328266 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:12.335825 20875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:12.335963 20875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:26:12.336030 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:12.336140 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:12.336238 20875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:12.336337 20875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:12.346689 20875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.126.keytab
--principal=oryx/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:12.348131 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:12.349576 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:12.362311 20880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:12.362406 20881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:12.362444 20883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:12.362782 20875 server_base.cc:1034] running on GCE node
I20260925 10:26:12.363904 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:12.365476 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:12.366761 20875 hybrid_clock.cc:648] HybridClock initialized: now 1790331972366682 us; error 34 us; skew 500 ppm
I20260925 10:26:12.367138 20875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:12 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:12.370334 20875 init.cc:377] Logged in from keytab as oryx/127.12.248.126@KRBTEST.COM (short username oryx)
I20260925 10:26:12.371841 20875 webserver.cc:460] Webserver started at http://127.12.248.126:34897/ using document root <none> and password file <none>
I20260925 10:26:12.372545 20875 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:12.372632 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:12.372913 20875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:12.375916 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "5800b6b92a6d4b3fa788d5f933c332b0"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "bb2ce31af930dcdbfce0e78f1b1be60c"
server_key_iv: "8cddf25669ae356b357608308dd79e74"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.376647 20875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "5800b6b92a6d4b3fa788d5f933c332b0"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "bb2ce31af930dcdbfce0e78f1b1be60c"
server_key_iv: "8cddf25669ae356b357608308dd79e74"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.381789 20875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 10:26:12.385347 20890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:12.386775 20875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:12.386924 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "5800b6b92a6d4b3fa788d5f933c332b0"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "bb2ce31af930dcdbfce0e78f1b1be60c"
server_key_iv: "8cddf25669ae356b357608308dd79e74"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.387107 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:12.414191 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:12.419219 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:12.419471 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:12.439841 20875 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37229
I20260925 10:26:12.439862 20951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37229 every 8 connection(s)
I20260925 10:26:12.441188 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:12.445093 20952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:12.449143 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20875
I20260925 10:26:12.449368 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:12.450062 13281 external_mini_cluster.cc:1428] Setting key 9106c930d31af6f1d6cacda53131cc26
I20260925 10:26:12.456234 20952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Bootstrap starting.
I20260925 10:26:12.460005 20952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:12.461138 20952 log.cc:824] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:12.464685 20952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: No bootstrap required, opened a new log
Sep 25 10:26:12 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:12.474782 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:12.452085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.12.248.126:37229)
0925 10:26:12.452867 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:12.452879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:12.452921 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:26:12.454721 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:12.454744 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:12.455137 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:12.455476 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:12.457258 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:12.458738 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:12.460489 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:12.461164 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:12.461318 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:12.466191 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:12.466203 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:12.466228 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:12.466284 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:12.470234 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:12.470883 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:12.470893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:12.470906 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:12.471120 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:12.471481 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:12.471486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:12.471489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:12.472190 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:12.472499 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:12.472787 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:26:12.473259 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":370,"threads_started":1}
I20260925 10:26:12.475566 20952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } }
I20260925 10:26:12.475937 20952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:12.476058 20952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5800b6b92a6d4b3fa788d5f933c332b0, State: Initialized, Role: FOLLOWER
I20260925 10:26:12.476730 20952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } }
I20260925 10:26:12.477000 20952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:12.477142 20952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:12.477329 20952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:12.479597 20952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } }
I20260925 10:26:12.480150 20952 leader_election.cc:302] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5800b6b92a6d4b3fa788d5f933c332b0; no voters:
I20260925 10:26:12.480718 20952 leader_election.cc:288] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:12.480877 20957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:12.481729 20957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5800b6b92a6d4b3fa788d5f933c332b0, State: Running, Role: LEADER
I20260925 10:26:12.482287 20957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } }
I20260925 10:26:12.483006 20952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:12.485172 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5800b6b92a6d4b3fa788d5f933c332b0. Latest consensus state: current_term: 1 leader_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } } }
I20260925 10:26:12.485663 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:12.485203 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800b6b92a6d4b3fa788d5f933c332b0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37229 } } }
I20260925 10:26:12.488420 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:12.489600 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:12.494964 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:12.503623 20963 catalog_manager.cc:1347] Generated new cluster ID: d6053477a13541a284689d4d5f855386
I20260925 10:26:12.503732 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:12.519059 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:12.520397 20963 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:12.527664 20963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Generated new TSK 0
I20260925 10:26:12.528710 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
I20260925 10:26:12.589784 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37229
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:12.922358 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:12.922691 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:12.922866 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:12.930205 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:12.930362 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:12.930500 20978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:26:12.930629 20978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:12.940740 20978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:12.942261 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:12.943648 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:12.957055 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:12.957108 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:12.957423 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:12.958279 20978 server_base.cc:1034] running on GCE node
I20260925 10:26:12.958874 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:12.959826 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:12.961082 20978 hybrid_clock.cc:648] HybridClock initialized: now 1790331972960973 us; error 73 us; skew 500 ppm
I20260925 10:26:12.961387 20978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:12 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:12.964437 20978 init.cc:377] Logged in from keytab as oryx/127.12.248.65@KRBTEST.COM (short username oryx)
I20260925 10:26:12.966014 20978 webserver.cc:460] Webserver started at http://127.12.248.65:42769/ using document root <none> and password file <none>
I20260925 10:26:12.966713 20978 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:12.966840 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:12.967170 20978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:12.970178 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "648aff6489e749c18f89af45bb156473"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "66a2e7ea03bc3ed9d66a4e04920974aa"
server_key_iv: "42fb9e241d98029fc259938703919f60"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.970953 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "648aff6489e749c18f89af45bb156473"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "66a2e7ea03bc3ed9d66a4e04920974aa"
server_key_iv: "42fb9e241d98029fc259938703919f60"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.976140 20978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:26:12.979766 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:12.981168 20978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:12.981353 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "648aff6489e749c18f89af45bb156473"
format_stamp: "Formatted at 2026-09-25 10:26:12 on dist-test-slave-f9rl"
server_key: "66a2e7ea03bc3ed9d66a4e04920974aa"
server_key_iv: "42fb9e241d98029fc259938703919f60"
server_key_version: "encryptionkey@0"
I20260925 10:26:12.981658 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:12.996814 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:13.001292 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:13.001641 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:13.004340 20978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:13.004460 20978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:13.004650 20978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:13.004742 20978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:26:13 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:13.028512 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.004893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:50059 (local address 127.12.248.126:37229)
0925 10:26:13.005841 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:13.005849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:13.006802 (+ 953us) server_negotiation.cc:407] Connection header received
0925 10:26:13.009052 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:13.009058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:13.009155 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:13.009270 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:13.011520 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.012563 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.015206 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.015844 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.015982 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.020971 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:13.020976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:13.020980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:13.021013 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:13.024411 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.025172 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.025179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.025185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.025352 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.026003 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.026007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.026011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.026374 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:13.026776 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:13.028053 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0925 10:26:13.028242 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":691,"threads_started":1}
I20260925 10:26:13.030982 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.005717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.65:50059)
0925 10:26:13.006610 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:13.006658 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:13.008764 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:13.009604 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:13.009620 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:13.010459 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:13.011328 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.011353 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.012713 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.012720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.014983 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.015005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.016168 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.016178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.016403 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.017340 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:13.017379 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:13.020712 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:13.024556 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.024566 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.024586 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.025025 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.025526 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.025532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.025535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.025791 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.026788 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:13.026803 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:13.027468 (+ 665us) client_negotiation.cc:769] Sending connection context
0925 10:26:13.027910 (+ 442us) client_negotiation.cc:240] Negotiation successful
0925 10:26:13.028696 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":380,"threads_started":1}
I20260925 10:26:13.051327 21113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:34449 every 8 connection(s)
I20260925 10:26:13.051326 20978 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:34449
I20260925 10:26:13.053386 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:13.055948 21114 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37229
I20260925 10:26:13.056221 21114 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:13.056821 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.039650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:47889 (local address 127.12.248.126:37229)
0925 10:26:13.039893 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:13.039902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:13.040491 (+ 589us) server_negotiation.cc:407] Connection header received
0925 10:26:13.040772 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:13.040778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:13.040886 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:13.041008 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:13.042504 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.043238 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.046095 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.046598 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.046747 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.049779 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:13.049784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:13.049787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:13.049820 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:13.053579 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.054504 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.054510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.054513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.054691 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.055185 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.055192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.055195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.055584 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:13.055749 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:13.056221 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 10:26:13.056457 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:26:13.056947 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.039480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.65:47889)
0925 10:26:13.040331 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:13.040373 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:13.040572 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:13.041040 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:13.041045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:13.041816 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:13.042350 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.042367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.043435 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.043445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.045899 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.045926 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.046859 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.046876 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.047162 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.048134 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:13.048165 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:13.049571 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:13.053729 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.053734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.053737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.054335 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.054871 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.054876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.054879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.055034 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.055716 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:13.055721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:13.055938 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 10:26:13.056089 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 10:26:13.056461 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":389,"threads_started":1}
I20260925 10:26:13.057120 21114 heartbeater.cc:511] Master 127.12.248.126:37229 requested a full tablet report, sending...
I20260925 10:26:13.059305 20907 ts_manager.cc:194] Registered new tserver with Master: 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449)
I20260925 10:26:13.061005 20907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:50059
I20260925 10:26:13.063716 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 20978
I20260925 10:26:13.063918 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:13.064610 13281 external_mini_cluster.cc:1428] Setting key 4c88cdc0299614f3fc40642eb8235e80
WARNING: no policy specified for oryx/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
I20260925 10:26:13.117208 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37229
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:13.440752 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:13.441066 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:13.441231 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:13.448716 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:13.448864 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:13.449018 21118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:26:13.449153 21118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:13.459213 21118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:13.460472 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:13.461985 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:13.474830 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:13.474860 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:13.475049 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:13.475600 21118 server_base.cc:1034] running on GCE node
I20260925 10:26:13.476164 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:13.477056 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:13.478274 21118 hybrid_clock.cc:648] HybridClock initialized: now 1790331973478214 us; error 54 us; skew 500 ppm
I20260925 10:26:13.478556 21118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:13 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:13.481745 21118 init.cc:377] Logged in from keytab as oryx/127.12.248.66@KRBTEST.COM (short username oryx)
I20260925 10:26:13.483286 21118 webserver.cc:460] Webserver started at http://127.12.248.66:39759/ using document root <none> and password file <none>
I20260925 10:26:13.483989 21118 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:13.484113 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:13.484421 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:13.487515 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "9fa4fd6bacb14071af4951d326998c99"
format_stamp: "Formatted at 2026-09-25 10:26:13 on dist-test-slave-f9rl"
server_key: "0cd4a99ca5cd3f289b934d8e23479e93"
server_key_iv: "86333eba55368978514650e294d6a19b"
server_key_version: "encryptionkey@0"
I20260925 10:26:13.488304 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "9fa4fd6bacb14071af4951d326998c99"
format_stamp: "Formatted at 2026-09-25 10:26:13 on dist-test-slave-f9rl"
server_key: "0cd4a99ca5cd3f289b934d8e23479e93"
server_key_iv: "86333eba55368978514650e294d6a19b"
server_key_version: "encryptionkey@0"
I20260925 10:26:13.494024 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:26:13.498896 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:13.500442 21118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260925 10:26:13.500633 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "9fa4fd6bacb14071af4951d326998c99"
format_stamp: "Formatted at 2026-09-25 10:26:13 on dist-test-slave-f9rl"
server_key: "0cd4a99ca5cd3f289b934d8e23479e93"
server_key_iv: "86333eba55368978514650e294d6a19b"
server_key_version: "encryptionkey@0"
I20260925 10:26:13.500866 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:13.523851 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:13.528527 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:13.528812 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:13.531690 21118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:13.531795 21118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:13.531937 21118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:13.532006 21118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:26:13 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331973, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:13.555986 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.532106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:48837 (local address 127.12.248.126:37229)
0925 10:26:13.532300 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:13.532310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:13.534119 (+ 1809us) server_negotiation.cc:407] Connection header received
0925 10:26:13.536173 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:13.536178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:13.536273 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:13.536381 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:13.538649 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.539667 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.542188 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.542949 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.543116 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.548472 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:13.548477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:13.548481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:13.548516 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:13.552191 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.552912 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.552921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.552927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.553133 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.553636 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.553644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.553658 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.554123 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:13.554306 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:13.554975 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 10:26:13.555288 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:26:13.556861 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.532884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.66:48837)
0925 10:26:13.533916 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:13.533976 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:13.535901 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:13.536689 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:13.536702 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:13.537582 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:13.538466 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.538487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.539822 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.539832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.542019 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.542041 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.543220 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.543231 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.543548 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.544404 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:13.544431 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:13.548148 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:13.552344 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.552351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.552363 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.552781 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.553259 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.553264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.553267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.553519 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.554247 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:13.554255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:13.554586 (+ 331us) client_negotiation.cc:769] Sending connection context
0925 10:26:13.554812 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 10:26:13.555204 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":454,"threads_started":1}
I20260925 10:26:13.579348 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.563014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.66:52407)
0925 10:26:13.563639 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:13.563661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:13.563773 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:13.564164 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:13.564167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:13.564648 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:13.564920 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.564928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.566784 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.566790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.568553 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:13.568565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.569834 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.569839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:13.569957 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.570635 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:13.570675 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:13.572127 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:13.576078 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.576083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.576087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.576751 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.577298 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:13.577304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:13.577308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:13.577554 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:13.578429 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:13.578433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:13.578597 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 10:26:13.578715 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:26:13.578964 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":337,"threads_started":1}
I20260925 10:26:13.579707 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:13.563089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:52407 (local address 127.12.248.126:37229)
0925 10:26:13.563325 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:13.563333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:13.563740 (+ 407us) server_negotiation.cc:407] Connection header received
0925 10:26:13.563889 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:13.563898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:13.563998 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:13.564149 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:13.565064 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.566482 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.568723 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:13.569641 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:13.569798 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:13.572403 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:13.572413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:13.572420 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:13.572455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:13.575950 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.576933 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.576941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.576948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.577120 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:13.577680 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:13.577688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:13.577694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:13.578288 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:13.578455 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:13.578872 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 10:26:13.579227 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:26:13.589790 21118 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:37809
I20260925 10:26:13.589819 21252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:37809 every 8 connection(s)
I20260925 10:26:13.591217 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:13.593297 21253 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37229
I20260925 10:26:13.593530 21253 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:13.594110 21253 heartbeater.cc:511] Master 127.12.248.126:37229 requested a full tablet report, sending...
I20260925 10:26:13.595369 20907 ts_manager.cc:194] Registered new tserver with Master: 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:13.596135 20907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:48837
I20260925 10:26:13.599207 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21118
I20260925 10:26:13.599386 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:13.599843 13281 external_mini_cluster.cc:1428] Setting key 26fe83b68fe71502b1b967a4096db4b9
WARNING: no policy specified for oryx/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
I20260925 10:26:13.648823 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37229
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:13.978169 21257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:13.978457 21257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:13.978641 21257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:13.986279 21257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:13.986467 21257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:13.986626 21257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:26:13.986773 21257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:13.996668 21257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.248.126:37229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:13.998030 21257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:13.999305 21257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:14.012308 21262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:14.012195 21263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:14.012274 21265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:14.012910 21257 server_base.cc:1034] running on GCE node
I20260925 10:26:14.013552 21257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:14.014432 21257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:14.015679 21257 hybrid_clock.cc:648] HybridClock initialized: now 1790331974015605 us; error 48 us; skew 500 ppm
I20260925 10:26:14.016000 21257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331974, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:14.019388 21257 init.cc:377] Logged in from keytab as oryx/127.12.248.67@KRBTEST.COM (short username oryx)
I20260925 10:26:14.020918 21257 webserver.cc:460] Webserver started at http://127.12.248.67:40137/ using document root <none> and password file <none>
I20260925 10:26:14.021705 21257 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:14.021842 21257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:14.022164 21257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:14.025120 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "5305461f474d43c798ff5cb998a9d8c9"
format_stamp: "Formatted at 2026-09-25 10:26:14 on dist-test-slave-f9rl"
server_key: "23c1fa18fec8782099d6f97fd124869e"
server_key_iv: "345af1e2201a683cf3bb5cd8d087e10f"
server_key_version: "encryptionkey@0"
I20260925 10:26:14.025960 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "5305461f474d43c798ff5cb998a9d8c9"
format_stamp: "Formatted at 2026-09-25 10:26:14 on dist-test-slave-f9rl"
server_key: "23c1fa18fec8782099d6f97fd124869e"
server_key_iv: "345af1e2201a683cf3bb5cd8d087e10f"
server_key_version: "encryptionkey@0"
I20260925 10:26:14.031148 21257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:26:14.034662 21272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:14.036079 21257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:26:14.036254 21257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "5305461f474d43c798ff5cb998a9d8c9"
format_stamp: "Formatted at 2026-09-25 10:26:14 on dist-test-slave-f9rl"
server_key: "23c1fa18fec8782099d6f97fd124869e"
server_key_iv: "345af1e2201a683cf3bb5cd8d087e10f"
server_key_version: "encryptionkey@0"
I20260925 10:26:14.036482 21257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:14.053736 21257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:14.058442 21257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:14.058732 21257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:14.061595 21257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:14.061713 21257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:14.061858 21257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:14.061947 21257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:14.063802 21114 heartbeater.cc:503] Master 127.12.248.126:37229 was elected leader, sending a full tablet report...
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331974, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:14.088554 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.062258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:45007 (local address 127.12.248.126:37229)
0925 10:26:14.062485 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.062492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.064452 (+ 1960us) server_negotiation.cc:407] Connection header received
0925 10:26:14.067030 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.067035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.067118 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.067232 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:14.068702 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.069743 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.073425 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.073880 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.074060 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.080747 (+ 6687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:14.080754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:14.080758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:14.080808 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:14.084161 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.085165 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.085173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.085177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.085352 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.085938 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.085944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.085950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.086456 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:14.086639 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.087568 (+ 929us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.087913 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:26:14.089227 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.063261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.67:45007)
0925 10:26:14.064265 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.064327 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.066763 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.067463 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.067471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.067959 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:14.068541 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.068558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.069971 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.069979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.073237 (+ 3258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.073265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.074187 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.074199 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.074644 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.075608 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:14.075653 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:14.080429 (+ 4776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:14.084293 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.084343 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.084371 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.085030 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.085497 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.085502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.085506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.085775 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.086639 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:14.086654 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:14.087077 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.087309 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.087763 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":465,"threads_started":1}
I20260925 10:26:14.112040 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.096325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:36149 (local address 127.12.248.126:37229)
0925 10:26:14.096577 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.096586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.097180 (+ 594us) server_negotiation.cc:407] Connection header received
0925 10:26:14.097321 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.097330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.097468 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.097614 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:14.099030 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.099751 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.101162 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.101659 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.101793 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.104473 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:14.104484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:14.104492 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:14.104551 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:14.109116 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.110071 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.110078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.110082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.110212 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.110576 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.110581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.110585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.110877 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:14.111036 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.111438 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.111660 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:26:14.112145 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.096427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.12.248.67:36149)
0925 10:26:14.097066 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.097091 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.097213 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.097648 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.097652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.098357 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:14.098871 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.098882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.099893 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.099898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.101009 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.101021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.101887 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.101893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.102082 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.102941 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:14.102959 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:14.104233 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:14.109289 (+ 5056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.109296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.109302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.109933 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.110332 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.110336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.110340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.110468 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.111002 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:14.111007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:14.111193 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.111337 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.111683 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":354,"threads_started":1}
I20260925 10:26:14.123499 21257 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:35997
I20260925 10:26:14.123520 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:35997 every 8 connection(s)
I20260925 10:26:14.124861 21257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:14.127085 21392 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37229
I20260925 10:26:14.127279 21392 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:14.127902 21392 heartbeater.cc:511] Master 127.12.248.126:37229 requested a full tablet report, sending...
I20260925 10:26:14.129163 20907 ts_manager.cc:194] Registered new tserver with Master: 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997)
I20260925 10:26:14.130010 20907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:45007
I20260925 10:26:14.130280 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21257
I20260925 10:26:14.130461 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:14.130936 13281 external_mini_cluster.cc:1428] Setting key 09ebd032d4e2520ab3fcd355fb0eacb4
I20260925 10:26:14.134123 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:26:14.185141 21002 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:14.172919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.12.248.126:37229)
0925 10:26:14.173242 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.173249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.173478 (+ 229us) server_negotiation.cc:407] Connection header received
0925 10:26:14.173718 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.173723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.173798 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.173904 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:14.175251 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.175981 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.177220 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.177729 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.177861 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.183473 (+ 5612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37508: BlockingRecv error: recv got EOF from 127.0.0.1:37508 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331974, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:14.201256 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.187468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.12.248.126:37229)
0925 10:26:14.187693 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.187700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.187925 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:26:14.188057 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.188061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.188196 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.188289 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:14.189535 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.190482 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.191886 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.192355 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.192464 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.196456 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:14.196461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:14.196464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:14.196499 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:14.198981 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.199602 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.199606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.199608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.199734 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.200034 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.200037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.200039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.200344 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:14.200541 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.200753 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.200922 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:26:14.205355 20907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:14.212343 20907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:14.249344 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.225387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37809 (local address 127.0.0.1:43958)
0925 10:26:14.240309 (+ 14922us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.240330 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.240434 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.241613 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.241618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.241659 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.242206 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.242218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243933 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.243947 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.246547 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.246559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248214 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.248431 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248489 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.248634 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.248839 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14777,"mutex_wait_us":448,"spinlock_wait_cycles":2304,"thread_start_us":14559,"threads_started":1}
I20260925 10:26:14.249392 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.225970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.12.248.66:37809)
0925 10:26:14.240687 (+ 14717us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.240698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.240728 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:26:14.240895 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.240901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.241129 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.241383 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.242404 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243789 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.246683 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248009 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248157 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248419 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.248944 (+ 525us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.249109 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14534,"thread_start_us":14420,"threads_started":1}
I20260925 10:26:14.249965 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.225041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.12.248.65:34449)
0925 10:26:14.240936 (+ 15895us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.240944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.240968 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:26:14.241046 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.241052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.241242 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.241414 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.242541 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243598 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.247283 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248276 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248409 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248612 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.249473 (+ 861us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.249667 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15718,"thread_start_us":15567,"threads_started":1}
I20260925 10:26:14.250393 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.225886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.12.248.67:35997)
0925 10:26:14.239197 (+ 13311us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.239210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.240301 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 10:26:14.240425 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.240435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.240713 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.240938 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.241758 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243612 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.247054 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248427 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248579 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248813 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.249884 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.250045 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13079,"thread_start_us":13696,"threads_started":1}
I20260925 10:26:14.249739 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.225043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35997 (local address 127.0.0.1:46812)
0925 10:26:14.239961 (+ 14918us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.239983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.240182 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.241035 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.241043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.241078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.241490 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.241503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243821 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.243860 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.246906 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.246918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248638 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.248825 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248870 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.248949 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.249089 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14521,"mutex_wait_us":923,"thread_start_us":12502,"threads_started":1}
I20260925 10:26:14.250885 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.224799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.0.0.1:52760)
0925 10:26:14.238107 (+ 13308us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.238164 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.238432 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.241423 (+ 2991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.241453 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.241481 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.242001 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.242014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.243729 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.243737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.247149 (+ 3412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.247159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.248772 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.248779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.248917 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.248955 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.249038 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.249168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12944,"mutex_wait_us":606,"spinlock_wait_cycles":4352,"thread_start_us":14910,"threads_started":1}
I20260925 10:26:14.253979 21044 tablet_service.cc:1467] Processing CreateTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (DEFAULT_TABLE table=test-table [id=0b4ec739bc20424fb7eb1de7353cc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.254045 21322 tablet_service.cc:1467] Processing CreateTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (DEFAULT_TABLE table=test-table [id=0b4ec739bc20424fb7eb1de7353cc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.254047 21183 tablet_service.cc:1467] Processing CreateTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (DEFAULT_TABLE table=test-table [id=0b4ec739bc20424fb7eb1de7353cc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.255376 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.255376 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.255402 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.266775 21416 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Bootstrap starting.
I20260925 10:26:14.269822 21417 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Bootstrap starting.
I20260925 10:26:14.271432 21416 tablet_bootstrap.cc:654] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.272766 21416 log.cc:824] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:14.273378 21417 tablet_bootstrap.cc:654] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.274220 21418 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Bootstrap starting.
I20260925 10:26:14.274663 21417 log.cc:824] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:14.277665 21417 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: No bootstrap required, opened a new log
I20260925 10:26:14.277665 21418 tablet_bootstrap.cc:654] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.277949 21417 ts_tablet_manager.cc:1397] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 10:26:14.278440 21416 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: No bootstrap required, opened a new log
I20260925 10:26:14.278698 21416 ts_tablet_manager.cc:1397] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260925 10:26:14.278867 21418 log.cc:824] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:14.281764 21418 tablet_bootstrap.cc:492] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: No bootstrap required, opened a new log
I20260925 10:26:14.282022 21418 ts_tablet_manager.cc:1397] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 10:26:14.286692 21416 raft_consensus.cc:348] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.287070 21416 raft_consensus.cc:374] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.287159 21416 raft_consensus.cc:729] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648aff6489e749c18f89af45bb156473, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.287955 21416 consensus_queue.cc:260] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.289129 21417 raft_consensus.cc:348] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.289654 21417 raft_consensus.cc:374] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.289769 21417 raft_consensus.cc:729] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.290650 21417 consensus_queue.cc:260] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.290953 21416 ts_tablet_manager.cc:1428] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260925 10:26:14.292131 21253 heartbeater.cc:503] Master 127.12.248.126:37229 was elected leader, sending a full tablet report...
I20260925 10:26:14.292850 21417 ts_tablet_manager.cc:1428] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 10:26:14.295115 21418 raft_consensus.cc:348] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.295600 21418 raft_consensus.cc:374] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.295750 21418 raft_consensus.cc:729] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.296370 21418 consensus_queue.cc:260] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.297573 21392 heartbeater.cc:503] Master 127.12.248.126:37229 was elected leader, sending a full tablet report...
I20260925 10:26:14.298134 21418 ts_tablet_manager.cc:1428] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260925 10:26:14.306313 21115 tablet.cc:2369] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:14.325011 21423 raft_consensus.cc:482] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:14.325273 21423 raft_consensus.cc:504] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.327013 21423 leader_election.cc:288] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997)
I20260925 10:26:14.333751 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.327466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:58129 (local address 127.12.248.67:35997)
0925 10:26:14.327655 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.327660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.328318 (+ 658us) server_negotiation.cc:407] Connection header received
0925 10:26:14.328529 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.328533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.328590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.328679 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.329289 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.330057 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.331966 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.332730 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.332923 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.333007 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.333335 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.333495 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:26:14.333761 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:44393 (local address 127.12.248.65:34449)
0925 10:26:14.327587 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.327593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.328317 (+ 724us) server_negotiation.cc:407] Connection header received
0925 10:26:14.328506 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.328510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.328579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.328671 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.329344 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.330105 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.331973 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.332738 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.332916 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.333007 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.333355 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.333507 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 10:26:14.333904 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.327482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35997 (local address 127.12.248.66:58129)
0925 10:26:14.328211 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.328225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.328396 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.328692 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.328697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.328743 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.329132 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.329141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.330185 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.330193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.331820 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.331831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.332990 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.333000 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.333143 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.333206 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.333308 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.333438 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":93,"spinlock_wait_cycles":32512,"thread_start_us":445,"threads_started":1}
I20260925 10:26:14.334184 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.327188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.12.248.66:44393)
0925 10:26:14.328183 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.328209 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.328362 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.328688 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.328692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.328737 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.329210 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.329220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.330244 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.330249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.331818 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.331830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.333015 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.333023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.333203 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.333226 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.333328 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.333511 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":18,"spinlock_wait_cycles":21760,"thread_start_us":521,"threads_started":1}
I20260925 10:26:14.334879 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c5051fbb5342fa9cdb39b9ac2b25e9" candidate_uuid: "9fa4fd6bacb14071af4951d326998c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9" is_pre_election: true
I20260925 10:26:14.335119 21064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c5051fbb5342fa9cdb39b9ac2b25e9" candidate_uuid: "9fa4fd6bacb14071af4951d326998c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473" is_pre_election: true
I20260925 10:26:14.335383 21347 raft_consensus.cc:2454] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa4fd6bacb14071af4951d326998c99 in term 0.
I20260925 10:26:14.335691 21064 raft_consensus.cc:2454] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa4fd6bacb14071af4951d326998c99 in term 0.
I20260925 10:26:14.336331 21136 leader_election.cc:302] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:14.336794 21423 raft_consensus.cc:2781] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:14.336951 21423 raft_consensus.cc:482] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:14.337064 21423 raft_consensus.cc:3037] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.339951 21423 raft_consensus.cc:504] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.340611 21423 leader_election.cc:288] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [CANDIDATE]: Term 1 election: Requested vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997)
I20260925 10:26:14.340981 21064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c5051fbb5342fa9cdb39b9ac2b25e9" candidate_uuid: "9fa4fd6bacb14071af4951d326998c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473"
I20260925 10:26:14.341171 21064 raft_consensus.cc:3037] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.341140 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c5051fbb5342fa9cdb39b9ac2b25e9" candidate_uuid: "9fa4fd6bacb14071af4951d326998c99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9"
I20260925 10:26:14.341369 21347 raft_consensus.cc:3037] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:26:14.343255 21254 tablet.cc:2369] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:14.343427 21064 raft_consensus.cc:2454] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa4fd6bacb14071af4951d326998c99 in term 1.
I20260925 10:26:14.343823 21135 leader_election.cc:302] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648aff6489e749c18f89af45bb156473, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:14.343997 21347 raft_consensus.cc:2454] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa4fd6bacb14071af4951d326998c99 in term 1.
I20260925 10:26:14.344105 21423 raft_consensus.cc:2781] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:14.344606 21423 raft_consensus.cc:686] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Running, Role: LEADER
I20260925 10:26:14.345109 21423 consensus_queue.cc:237] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.349648 20906 catalog_manager.cc:5495] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "9fa4fd6bacb14071af4951d326998c99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } health_report { overall_health: HEALTHY } } }
I20260925 10:26:14.372375 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.367594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.12.248.66:37809)
0925 10:26:14.367799 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.367804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.367821 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:14.368009 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.368013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.368117 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.368243 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:14.368550 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.369267 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.370504 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.370930 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.371022 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.371382 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:14.371933 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:14.372036 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.372111 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.372174 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:26:14.376631 21393 tablet.cc:2369] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:14.380565 21347 raft_consensus.cc:1261] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Refusing update from remote peer 9fa4fd6bacb14071af4951d326998c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:14.380571 21064 raft_consensus.cc:1261] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Refusing update from remote peer 9fa4fd6bacb14071af4951d326998c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:14.381508 21427 consensus_queue.cc:1035] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:14.381829 21423 consensus_queue.cc:1035] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:14.396767 21430 mvcc.cc:205] Tried to move back new op lower bound from 7333199767053844480 to 7333199766921715712. Current Snapshot: MvccSnapshot[applied={T|T < 7333199767053844480}]
I20260925 10:26:14.399236 21432 mvcc.cc:205] Tried to move back new op lower bound from 7333199767053844480 to 7333199766921715712. Current Snapshot: MvccSnapshot[applied={T|T < 7333199767053844480}]
I20260925 10:26:14.400692 21431 mvcc.cc:205] Tried to move back new op lower bound from 7333199767053844480 to 7333199766921715712. Current Snapshot: MvccSnapshot[applied={T|T < 7333199767053844480}]
I20260925 10:26:14.410471 20907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:14.410756 20907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:14.414309 20907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Sending DeleteTablet for 3 replicas of tablet 65c5051fbb5342fa9cdb39b9ac2b25e9
I20260925 10:26:14.415509 21322 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:14 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46812
I20260925 10:26:14.415680 21183 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:14 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260925 10:26:14.416249 21044 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:14 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:52760
I20260925 10:26:14.417042 20907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:14.417970 20907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:14.424989 21439 tablet_replica.cc:330] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: stopping tablet replica
I20260925 10:26:14.427444 21182 tablet_service.cc:1467] Processing CreateTablet for tablet 31dfe48405a740df87f220500e8e7e45 (DEFAULT_TABLE table=test-table [id=b409b4509da64d0fbcced92cfd0c4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.428052 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31dfe48405a740df87f220500e8e7e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.427944 21042 tablet_service.cc:1467] Processing CreateTablet for tablet 31dfe48405a740df87f220500e8e7e45 (DEFAULT_TABLE table=test-table [id=b409b4509da64d0fbcced92cfd0c4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.428139 21441 tablet_replica.cc:330] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: stopping tablet replica
I20260925 10:26:14.428535 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31dfe48405a740df87f220500e8e7e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.428604 21441 raft_consensus.cc:2229] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:14.429138 21441 raft_consensus.cc:2258] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:14.427865 21321 tablet_service.cc:1467] Processing CreateTablet for tablet 31dfe48405a740df87f220500e8e7e45 (DEFAULT_TABLE table=test-table [id=b409b4509da64d0fbcced92cfd0c4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:14.430073 21321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31dfe48405a740df87f220500e8e7e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.435061 21416 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Bootstrap starting.
I20260925 10:26:14.435086 21417 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Bootstrap starting.
I20260925 10:26:14.436681 21416 tablet_bootstrap.cc:654] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.437304 21417 tablet_bootstrap.cc:654] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.443943 21418 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Bootstrap starting.
I20260925 10:26:14.446168 21418 tablet_bootstrap.cc:654] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.425612 21439 raft_consensus.cc:2229] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:14.448099 21439 raft_consensus.cc:2258] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:14.451122 21441 ts_tablet_manager.cc:1905] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:14.454983 21439 ts_tablet_manager.cc:1905] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:14.462174 21417 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: No bootstrap required, opened a new log
I20260925 10:26:14.462324 21417 ts_tablet_manager.cc:1397] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20260925 10:26:14.463161 21417 raft_consensus.cc:348] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.463379 21417 raft_consensus.cc:374] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.463457 21417 raft_consensus.cc:729] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.463719 21417 consensus_queue.cc:260] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.467013 21417 ts_tablet_manager.cc:1428] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 10:26:14.469507 21418 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: No bootstrap required, opened a new log
I20260925 10:26:14.469663 21418 ts_tablet_manager.cc:1397] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20260925 10:26:14.470520 21418 raft_consensus.cc:348] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.470757 21418 raft_consensus.cc:374] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.470880 21418 raft_consensus.cc:729] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.471268 21440 tablet_replica.cc:330] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: stopping tablet replica
I20260925 10:26:14.471156 21418 consensus_queue.cc:260] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.471746 21418 ts_tablet_manager.cc:1428] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 10:26:14.471738 21440 raft_consensus.cc:2229] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:14.472572 21440 raft_consensus.cc:2258] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:14.476749 21416 tablet_bootstrap.cc:492] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: No bootstrap required, opened a new log
I20260925 10:26:14.476933 21416 ts_tablet_manager.cc:1397] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.016s
I20260925 10:26:14.477164 21440 ts_tablet_manager.cc:1905] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:14.477782 21416 raft_consensus.cc:348] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.478018 21416 raft_consensus.cc:374] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.478119 21416 raft_consensus.cc:729] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648aff6489e749c18f89af45bb156473, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.478430 21416 consensus_queue.cc:260] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.481179 21416 ts_tablet_manager.cc:1428] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:26:14.481554 21439 ts_tablet_manager.cc:1918] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:14.481716 21439 log.cc:1196] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/65c5051fbb5342fa9cdb39b9ac2b25e9
I20260925 10:26:14.482230 21439 ts_tablet_manager.cc:1939] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 5305461f474d43c798ff5cb998a9d8c9: Deleting consensus metadata
I20260925 10:26:14.484063 20893 catalog_manager.cc:4841] TS 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997): tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (table test-table [id=0b4ec739bc20424fb7eb1de7353cc441]) successfully deleted
I20260925 10:26:14.487012 21441 ts_tablet_manager.cc:1918] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:14.487185 21441 log.cc:1196] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/65c5051fbb5342fa9cdb39b9ac2b25e9
I20260925 10:26:14.487736 21441 ts_tablet_manager.cc:1939] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 648aff6489e749c18f89af45bb156473: Deleting consensus metadata
I20260925 10:26:14.489571 20892 catalog_manager.cc:4841] TS 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449): tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (table test-table [id=0b4ec739bc20424fb7eb1de7353cc441]) successfully deleted
I20260925 10:26:14.490118 21440 ts_tablet_manager.cc:1918] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:14.490306 21440 log.cc:1196] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/65c5051fbb5342fa9cdb39b9ac2b25e9
I20260925 10:26:14.490911 21440 ts_tablet_manager.cc:1939] T 65c5051fbb5342fa9cdb39b9ac2b25e9 P 9fa4fd6bacb14071af4951d326998c99: Deleting consensus metadata
I20260925 10:26:14.492519 20894 catalog_manager.cc:4841] TS 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809): tablet 65c5051fbb5342fa9cdb39b9ac2b25e9 (table test-table [id=0b4ec739bc20424fb7eb1de7353cc441]) successfully deleted
I20260925 10:26:14.571797 21424 raft_consensus.cc:482] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:14.572088 21424 raft_consensus.cc:504] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.574020 21424 leader_election.cc:288] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:14.580387 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.574226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.12.248.67:55127)
0925 10:26:14.574425 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.574445 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.574589 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.575104 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.575109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.575139 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.575375 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.575386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.576882 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.576887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.578523 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.578538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.579632 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.579638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.579854 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.579883 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.579992 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.580123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 10:26:14.580415 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.574561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:55127 (local address 127.12.248.65:34449)
0925 10:26:14.574721 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.574727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.574745 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:26:14.574826 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.574832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.574936 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.575061 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.575564 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.576737 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.578681 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.579473 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.579645 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.579722 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.580018 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.580147 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:26:14.581022 21063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31dfe48405a740df87f220500e8e7e45" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473" is_pre_election: true
I20260925 10:26:14.581323 21063 raft_consensus.cc:2454] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 0.
I20260925 10:26:14.581869 21274 leader_election.cc:302] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 648aff6489e749c18f89af45bb156473; no voters:
I20260925 10:26:14.582379 21424 raft_consensus.cc:2781] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:14.582501 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.574469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38417 (local address 127.12.248.66:37809)
0925 10:26:14.574644 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.574651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.575769 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 10:26:14.575934 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.575938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.576017 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.576110 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.576605 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.577677 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.580162 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.581457 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.581621 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.581691 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.582054 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.582196 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:26:14.582512 21424 raft_consensus.cc:482] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:14.582569 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.574226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37809 (local address 127.12.248.67:38417)
0925 10:26:14.575643 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.575657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.575789 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.576123 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.576127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.576149 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.576462 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.576470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.577818 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.577826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.579981 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.579996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.581680 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.581686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.581874 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.581913 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.582015 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.582134 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":437,"thread_start_us":703,"threads_started":1}
I20260925 10:26:14.582638 21424 raft_consensus.cc:3037] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.583626 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31dfe48405a740df87f220500e8e7e45" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99" is_pre_election: true
I20260925 10:26:14.584082 21208 raft_consensus.cc:2454] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 0.
I20260925 10:26:14.585391 21424 raft_consensus.cc:504] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.586091 21424 leader_election.cc:288] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 election: Requested vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:14.586606 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31dfe48405a740df87f220500e8e7e45" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99"
I20260925 10:26:14.586611 21063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31dfe48405a740df87f220500e8e7e45" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473"
I20260925 10:26:14.586812 21208 raft_consensus.cc:3037] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.586822 21063 raft_consensus.cc:3037] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.589277 21063 raft_consensus.cc:2454] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 1.
I20260925 10:26:14.590490 21274 leader_election.cc:302] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 648aff6489e749c18f89af45bb156473; no voters:
I20260925 10:26:14.591399 21208 raft_consensus.cc:2454] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 1.
I20260925 10:26:14.593672 21424 raft_consensus.cc:2781] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:14.594116 21424 raft_consensus.cc:686] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Running, Role: LEADER
I20260925 10:26:14.594830 21424 consensus_queue.cc:237] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:14.598718 20906 catalog_manager.cc:5495] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "5305461f474d43c798ff5cb998a9d8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } health_report { overall_health: HEALTHY } } }
Sep 25 10:26:14 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331972, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.126@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:14.672817 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.651911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.12.248.126:37229)
0925 10:26:14.652115 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.652120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.652830 (+ 710us) server_negotiation.cc:407] Connection header received
0925 10:26:14.653032 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.653037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.653100 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.653202 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:14.654419 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.655179 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.656428 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.656863 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.656995 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.667190 (+ 10195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:14.667195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:14.667199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:14.667229 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:14.669871 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.670793 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.670796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.670799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.670906 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:14.671310 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:14.671313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:14.671315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:14.671571 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:14.671761 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.672134 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.672465 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":91}
I20260925 10:26:14.672839 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.652074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37229 (local address 127.0.0.1:37534)
0925 10:26:14.652702 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.652722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.652906 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.653220 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.653224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.653988 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:14.654275 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.654286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.655316 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.655324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.656280 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.656291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.657033 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.657039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.657196 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.664341 (+ 7145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:14.664384 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:14.666970 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:14.670019 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.670034 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.670043 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.670659 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.671012 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:14.671017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:14.671020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:14.671206 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:14.671708 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:14.671720 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:14.671889 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.672050 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.672346 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":352,"threads_started":1}
I20260925 10:26:14.678578 20906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:37534:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:26:14.679594 20906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:14.686868 21321 tablet_service.cc:1467] Processing CreateTablet for tablet b66b87d1d4fa4f9db9f748e6dcd78c71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15873ed98c6646549df5dc557d93d777]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:26:14.686897 21042 tablet_service.cc:1467] Processing CreateTablet for tablet b66b87d1d4fa4f9db9f748e6dcd78c71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15873ed98c6646549df5dc557d93d777]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:26:14.687352 21321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66b87d1d4fa4f9db9f748e6dcd78c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.687402 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66b87d1d4fa4f9db9f748e6dcd78c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.690369 21183 tablet_service.cc:1467] Processing CreateTablet for tablet b66b87d1d4fa4f9db9f748e6dcd78c71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15873ed98c6646549df5dc557d93d777]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:26:14.690956 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66b87d1d4fa4f9db9f748e6dcd78c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:14.693423 21418 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: Bootstrap starting.
I20260925 10:26:14.695112 21418 tablet_bootstrap.cc:654] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.695979 21416 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: Bootstrap starting.
I20260925 10:26:14.697726 21416 tablet_bootstrap.cc:654] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.701381 21417 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: Bootstrap starting.
I20260925 10:26:14.702808 21418 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: No bootstrap required, opened a new log
I20260925 10:26:14.703029 21418 ts_tablet_manager.cc:1397] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 10:26:14.703212 21417 tablet_bootstrap.cc:654] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:14.704015 21418 raft_consensus.cc:348] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.704491 21418 raft_consensus.cc:374] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.704643 21418 raft_consensus.cc:729] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.705061 21418 consensus_queue.cc:260] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.705981 21418 ts_tablet_manager.cc:1428] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:14.705897 21424 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:14.706909 21424 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:14.708508 21417 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: No bootstrap required, opened a new log
I20260925 10:26:14.708719 21417 ts_tablet_manager.cc:1397] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 10:26:14.709792 21417 raft_consensus.cc:348] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.710114 21417 raft_consensus.cc:374] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.710218 21417 raft_consensus.cc:729] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.710585 21417 consensus_queue.cc:260] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.711362 21417 ts_tablet_manager.cc:1428] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:14.711161 21427 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:14.712159 21427 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:14.715160 21416 tablet_bootstrap.cc:492] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: No bootstrap required, opened a new log
I20260925 10:26:14.715371 21416 ts_tablet_manager.cc:1397] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
I20260925 10:26:14.716251 21416 raft_consensus.cc:348] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.716530 21416 raft_consensus.cc:374] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:14.716646 21416 raft_consensus.cc:729] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648aff6489e749c18f89af45bb156473, State: Initialized, Role: FOLLOWER
I20260925 10:26:14.716944 21416 consensus_queue.cc:260] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.717507 21422 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:14.717969 21422 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:14.718246 21416 ts_tablet_manager.cc:1428] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:26:14.930945 21422 raft_consensus.cc:482] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:14.931178 21422 raft_consensus.cc:504] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.932715 21422 leader_election.cc:288] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:14.941690 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.933154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:55247 (local address 127.12.248.67:35997)
0925 10:26:14.934123 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.934129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.934386 (+ 257us) server_negotiation.cc:407] Connection header received
0925 10:26:14.934491 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.934496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.934633 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.934727 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.935433 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.936476 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.939178 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.940442 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.940596 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.940669 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.941277 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.941414 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":707,"threads_started":1}
I20260925 10:26:14.941694 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.933181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:33013 (local address 127.12.248.66:37809)
0925 10:26:14.933415 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:14.933425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:14.933985 (+ 560us) server_negotiation.cc:407] Connection header received
0925 10:26:14.934305 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:14.934309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:14.934400 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:14.934505 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:14.935142 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.936262 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.939183 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.940423 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.940596 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.940669 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:14.941254 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 10:26:14.941402 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:26:14.941864 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.933148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37809 (local address 127.12.248.65:33013)
0925 10:26:14.933866 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.933889 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.934124 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.934555 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.934559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.934594 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.934967 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.934979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.936406 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.936410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.939020 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.939035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.940689 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.940695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.940958 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.941090 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.941192 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.941380 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":304,"thread_start_us":897,"threads_started":1}
I20260925 10:26:14.942101 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:14.933148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35997 (local address 127.12.248.65:55247)
0925 10:26:14.934299 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:14.934312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:14.934398 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:14.934758 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:14.934767 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:14.934795 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:14.935211 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.935224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.936595 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.936600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.939020 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:14.939035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:14.940660 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:14.940672 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:14.940961 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:14.941107 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:26:14.941192 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:26:14.941380 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":208,"thread_start_us":346,"threads_started":1}
I20260925 10:26:14.942508 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66b87d1d4fa4f9db9f748e6dcd78c71" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99" is_pre_election: true
I20260925 10:26:14.942826 21208 raft_consensus.cc:2454] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 0.
I20260925 10:26:14.942767 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66b87d1d4fa4f9db9f748e6dcd78c71" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9" is_pre_election: true
I20260925 10:26:14.943008 21347 raft_consensus.cc:2454] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 0.
I20260925 10:26:14.943301 20997 leader_election.cc:302] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648aff6489e749c18f89af45bb156473, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:14.943722 21422 raft_consensus.cc:2781] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:14.943866 21422 raft_consensus.cc:482] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:14.943976 21422 raft_consensus.cc:3037] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.946262 21422 raft_consensus.cc:504] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.946871 21422 leader_election.cc:288] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 election: Requested vote from peers 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:14.947352 21208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66b87d1d4fa4f9db9f748e6dcd78c71" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99"
I20260925 10:26:14.947376 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66b87d1d4fa4f9db9f748e6dcd78c71" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9"
I20260925 10:26:14.947554 21347 raft_consensus.cc:3037] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.947572 21208 raft_consensus.cc:3037] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:14.949510 21208 raft_consensus.cc:2454] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 1.
I20260925 10:26:14.949576 21347 raft_consensus.cc:2454] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 1.
I20260925 10:26:14.949905 20997 leader_election.cc:302] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648aff6489e749c18f89af45bb156473, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:14.950179 21422 raft_consensus.cc:2781] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:14.950538 21422 raft_consensus.cc:686] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [term 1 LEADER]: Becoming Leader. State: Replica: 648aff6489e749c18f89af45bb156473, State: Running, Role: LEADER
I20260925 10:26:14.950928 21422 consensus_queue.cc:237] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } }
I20260925 10:26:14.952826 21460 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: TxnStatusTablet state changed. Reason: New leader 648aff6489e749c18f89af45bb156473. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:14.953251 21460 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:26:14.954273 21462 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:26:14.954501 21462 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:26:14.954649 20907 catalog_manager.cc:5495] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 reported cstate change: term changed from 0 to 1, leader changed from <none> to 648aff6489e749c18f89af45bb156473 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } health_report { overall_health: UNKNOWN } } }
I20260925 10:26:15.006424 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.000482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.12.248.65:34449)
0925 10:26:15.000661 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.000669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.000687 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:26:15.000992 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.000996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.001090 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.001243 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:15.001672 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.002484 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.003849 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.004386 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.004542 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.005062 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.005705 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.005900 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.005987 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.006075 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:26:15.006438 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.000361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.0.0.1:52766)
0925 10:26:15.000601 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:15.000621 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:15.000768 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:15.001259 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:15.001264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:15.001276 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:26:15.001524 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.001535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.002637 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.002643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.003686 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.003698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.004662 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.004666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.004798 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.004857 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.005851 (+ 994us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.005877 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:26:15.005968 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:26:15.006125 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:26:15.010989 21347 raft_consensus.cc:1261] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Refusing update from remote peer 648aff6489e749c18f89af45bb156473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.011153 21208 raft_consensus.cc:1261] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Refusing update from remote peer 648aff6489e749c18f89af45bb156473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.011801 21422 consensus_queue.cc:1035] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.012128 21460 consensus_queue.cc:1035] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.019212 21424 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: TxnStatusTablet state changed. Reason: New leader 648aff6489e749c18f89af45bb156473. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.019625 21424 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:15.021971 21427 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: TxnStatusTablet state changed. Reason: New leader 648aff6489e749c18f89af45bb156473. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.022289 21427 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:15.024837 21422 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.025132 21422 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:26:15.029577 21460 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.029845 21460 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 648aff6489e749c18f89af45bb156473: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:26:15.036818 21424 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.037194 21424 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 5305461f474d43c798ff5cb998a9d8c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:15.048365 21427 tablet_replica.cc:439] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } }
I20260925 10:26:15.048657 21427 tablet_replica.cc:442] T b66b87d1d4fa4f9db9f748e6dcd78c71 P 9fa4fd6bacb14071af4951d326998c99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:26:15.057714 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.045639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.12.248.67:35997)
0925 10:26:15.045887 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.045894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.052146 (+ 6252us) server_negotiation.cc:407] Connection header received
0925 10:26:15.052410 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.052415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.052514 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.052681 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:15.053472 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.054181 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.055528 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.055949 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.056086 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.056493 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.057050 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.057210 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.057307 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.057393 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:26:15.079849 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.067903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:39705 (local address 127.12.248.65:34449)
0925 10:26:15.068172 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.068182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.073425 (+ 5243us) server_negotiation.cc:407] Connection header received
0925 10:26:15.074201 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.074209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.074308 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.074397 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:15.075092 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.076318 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.077898 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.078486 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.078641 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.079070 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.079182 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.079284 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.079419 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.079494 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:26:15.079901 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.067768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.12.248.67:39705)
0925 10:26:15.073317 (+ 5549us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:15.073333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:15.073524 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:15.074555 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:15.074559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:15.074573 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:26:15.074891 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.074903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.076499 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.076504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.077761 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.077773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.078675 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.078682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.078829 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.078883 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.079289 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.079303 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:26:15.079396 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:26:15.079529 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"thread_start_us":5250,"threads_started":1}
I20260925 10:26:15.096312 21207 raft_consensus.cc:1261] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Refusing update from remote peer 5305461f474d43c798ff5cb998a9d8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.096516 21063 raft_consensus.cc:1261] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Refusing update from remote peer 5305461f474d43c798ff5cb998a9d8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.097044 21424 consensus_queue.cc:1035] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.097335 21446 consensus_queue.cc:1035] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.155525 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.135290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:50175 (local address 127.12.248.67:35997)
0925 10:26:15.135715 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.135724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.149301 (+ 13577us) server_negotiation.cc:407] Connection header received
0925 10:26:15.149570 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.149576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.149693 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.149793 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:15.150408 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.151223 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.153650 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.154177 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.154250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.154761 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.154882 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.154983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.155116 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.155202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260925 10:26:15.155774 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.135353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35997 (local address 127.12.248.65:50175)
0925 10:26:15.149190 (+ 13837us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:15.149207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:15.149387 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:15.149810 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:15.149814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:15.149826 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:26:15.150239 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.150250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.151351 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.151357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.153470 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:15.153483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.154397 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.154401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:15.154520 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.154577 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.154989 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.155003 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:26:15.155095 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:26:15.155240 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13649,"thread_start_us":13371,"threads_started":1}
I20260925 10:26:15.210772 20906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:15.211089 20906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:15.214946 20906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Sending DeleteTablet for 3 replicas of tablet 31dfe48405a740df87f220500e8e7e45
I20260925 10:26:15.216048 21183 tablet_service.cc:1514] Processing DeleteTablet for tablet 31dfe48405a740df87f220500e8e7e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260925 10:26:15.216218 21321 tablet_service.cc:1514] Processing DeleteTablet for tablet 31dfe48405a740df87f220500e8e7e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46812
I20260925 10:26:15.220566 21042 tablet_service.cc:1514] Processing DeleteTablet for tablet 31dfe48405a740df87f220500e8e7e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:52760
I20260925 10:26:15.228716 21485 tablet_replica.cc:330] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: stopping tablet replica
I20260925 10:26:15.228963 21485 raft_consensus.cc:2229] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:15.229213 21485 raft_consensus.cc:2258] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.230499 21485 ts_tablet_manager.cc:1905] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.237792 21485 ts_tablet_manager.cc:1918] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:15.237939 21485 log.cc:1196] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/31dfe48405a740df87f220500e8e7e45
I20260925 10:26:15.238407 21485 ts_tablet_manager.cc:1939] T 31dfe48405a740df87f220500e8e7e45 P 9fa4fd6bacb14071af4951d326998c99: Deleting consensus metadata
I20260925 10:26:15.239609 20894 catalog_manager.cc:4841] TS 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809): tablet 31dfe48405a740df87f220500e8e7e45 (table test-table [id=b409b4509da64d0fbcced92cfd0c4106]) successfully deleted
I20260925 10:26:15.251937 21487 tablet_replica.cc:330] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: stopping tablet replica
I20260925 10:26:15.252226 21487 raft_consensus.cc:2229] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:15.252527 21487 raft_consensus.cc:2258] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.254040 21487 ts_tablet_manager.cc:1905] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.258949 21486 tablet_replica.cc:330] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: stopping tablet replica
I20260925 10:26:15.259200 21486 raft_consensus.cc:2229] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:15.260136 21486 raft_consensus.cc:2258] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.261317 21486 ts_tablet_manager.cc:1905] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.264675 21487 ts_tablet_manager.cc:1918] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:15.264847 21487 log.cc:1196] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/31dfe48405a740df87f220500e8e7e45
I20260925 10:26:15.265424 21487 ts_tablet_manager.cc:1939] T 31dfe48405a740df87f220500e8e7e45 P 648aff6489e749c18f89af45bb156473: Deleting consensus metadata
I20260925 10:26:15.269929 20892 catalog_manager.cc:4841] TS 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449): tablet 31dfe48405a740df87f220500e8e7e45 (table test-table [id=b409b4509da64d0fbcced92cfd0c4106]) successfully deleted
I20260925 10:26:15.270787 21486 ts_tablet_manager.cc:1918] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:15.270937 21486 log.cc:1196] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/31dfe48405a740df87f220500e8e7e45
I20260925 10:26:15.271489 21486 ts_tablet_manager.cc:1939] T 31dfe48405a740df87f220500e8e7e45 P 5305461f474d43c798ff5cb998a9d8c9: Deleting consensus metadata
I20260925 10:26:15.272720 20893 catalog_manager.cc:4841] TS 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997): tablet 31dfe48405a740df87f220500e8e7e45 (table test-table [id=b409b4509da64d0fbcced92cfd0c4106]) successfully deleted
Sep 25 10:26:15 dist-test-slave-f9rl krb5kdc[20862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:26:15 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:15 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:26:15.333308 21495 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:15.306296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.12.248.126:37229)
0925 10:26:15.318646 (+ 12350us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.318657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.319427 (+ 770us) server_negotiation.cc:407] Connection header received
0925 10:26:15.319643 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.319650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.319764 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.319892 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:15.322091 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.323283 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.325381 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.326126 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.326270 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.332944 (+ 6674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37546: BlockingRecv error: recv got EOF from 127.0.0.1:37546 (error 108)
Metrics: {"server-negotiator.queue_time_us":12166,"thread_start_us":12366,"threads_started":1}
Sep 25 10:26:15 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM
I20260925 10:26:15.357940 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.338234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.12.248.126:37229)
0925 10:26:15.338433 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.338440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.338459 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:26:15.338520 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.338525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.338600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.338703 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:15.340533 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.341749 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.343843 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.344510 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.344641 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.350448 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:15.350456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:15.350462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:15.350515 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:15.354659 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:15.355509 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:15.355516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:15.355520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:15.355740 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:15.356250 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:15.356256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:15.356260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:15.356629 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:15.356895 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.357281 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.357558 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:26:15.362910 20906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:15.363732 20906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:15.371945 21183 tablet_service.cc:1467] Processing CreateTablet for tablet 926151bf77eb471a8f658e07345f4b5c (DEFAULT_TABLE table=test-table [id=b7c0e0d1a84740debfd9c76147645e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.371979 21321 tablet_service.cc:1467] Processing CreateTablet for tablet 926151bf77eb471a8f658e07345f4b5c (DEFAULT_TABLE table=test-table [id=b7c0e0d1a84740debfd9c76147645e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.372457 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926151bf77eb471a8f658e07345f4b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.372468 21321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926151bf77eb471a8f658e07345f4b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.380554 21042 tablet_service.cc:1467] Processing CreateTablet for tablet 926151bf77eb471a8f658e07345f4b5c (DEFAULT_TABLE table=test-table [id=b7c0e0d1a84740debfd9c76147645e35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.381156 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 926151bf77eb471a8f658e07345f4b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.384222 21501 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Bootstrap starting.
I20260925 10:26:15.386583 21501 tablet_bootstrap.cc:654] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.392589 21504 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Bootstrap starting.
I20260925 10:26:15.392589 21502 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Bootstrap starting.
I20260925 10:26:15.395290 21504 tablet_bootstrap.cc:654] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.395924 21502 tablet_bootstrap.cc:654] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.398172 21501 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: No bootstrap required, opened a new log
I20260925 10:26:15.398388 21501 ts_tablet_manager.cc:1397] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260925 10:26:15.399278 21501 raft_consensus.cc:348] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.399658 21501 raft_consensus.cc:374] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.399814 21501 raft_consensus.cc:729] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.400151 21501 consensus_queue.cc:260] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.402976 21501 ts_tablet_manager.cc:1428] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 10:26:15.406339 21502 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: No bootstrap required, opened a new log
I20260925 10:26:15.406512 21502 ts_tablet_manager.cc:1397] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20260925 10:26:15.407424 21502 raft_consensus.cc:348] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.407732 21502 raft_consensus.cc:374] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.407852 21502 raft_consensus.cc:729] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.408149 21502 consensus_queue.cc:260] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.411617 21502 ts_tablet_manager.cc:1428] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260925 10:26:15.414635 21504 tablet_bootstrap.cc:492] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: No bootstrap required, opened a new log
I20260925 10:26:15.414876 21504 ts_tablet_manager.cc:1397] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20260925 10:26:15.415805 21504 raft_consensus.cc:348] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.416095 21504 raft_consensus.cc:374] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.416206 21504 raft_consensus.cc:729] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648aff6489e749c18f89af45bb156473, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.416513 21504 consensus_queue.cc:260] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.417111 21504 ts_tablet_manager.cc:1428] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:26:15.664050 21446 raft_consensus.cc:482] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:15.664309 21446 raft_consensus.cc:504] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.665015 21446 leader_election.cc:288] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:15.665639 21063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926151bf77eb471a8f658e07345f4b5c" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473" is_pre_election: true
I20260925 10:26:15.665726 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926151bf77eb471a8f658e07345f4b5c" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99" is_pre_election: true
I20260925 10:26:15.665884 21063 raft_consensus.cc:2454] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 0.
I20260925 10:26:15.666016 21207 raft_consensus.cc:2454] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 0.
I20260925 10:26:15.666450 21274 leader_election.cc:302] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 648aff6489e749c18f89af45bb156473; no voters:
I20260925 10:26:15.666795 21446 raft_consensus.cc:2781] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:15.666894 21446 raft_consensus.cc:482] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:15.667014 21446 raft_consensus.cc:3037] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.670367 21446 raft_consensus.cc:504] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.671128 21446 leader_election.cc:288] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 election: Requested vote from peers 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449), 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809)
I20260925 10:26:15.671638 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926151bf77eb471a8f658e07345f4b5c" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99"
I20260925 10:26:15.671663 21063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "926151bf77eb471a8f658e07345f4b5c" candidate_uuid: "5305461f474d43c798ff5cb998a9d8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "648aff6489e749c18f89af45bb156473"
I20260925 10:26:15.671873 21063 raft_consensus.cc:3037] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.671902 21207 raft_consensus.cc:3037] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.674316 21207 raft_consensus.cc:2454] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 1.
I20260925 10:26:15.674706 21063 raft_consensus.cc:2454] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5305461f474d43c798ff5cb998a9d8c9 in term 1.
I20260925 10:26:15.674839 21276 leader_election.cc:302] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:15.675148 21446 raft_consensus.cc:2781] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:15.675428 21446 raft_consensus.cc:686] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Running, Role: LEADER
I20260925 10:26:15.675717 21446 consensus_queue.cc:237] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.679368 20905 catalog_manager.cc:5495] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "5305461f474d43c798ff5cb998a9d8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } health_report { overall_health: HEALTHY } } }
I20260925 10:26:15.758600 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:15.753919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.12.248.67:35997)
0925 10:26:15.754472 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:15.754477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:15.754493 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:26:15.754551 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:15.754554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:15.754615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:15.754700 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:15.755038 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.755770 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.757019 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:15.757472 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:15.757578 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:15.757982 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:15.758080 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:15.758146 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:15.758266 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 10:26:15.758348 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":301,"threads_started":1}
I20260925 10:26:15.761773 21207 raft_consensus.cc:1261] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Refusing update from remote peer 5305461f474d43c798ff5cb998a9d8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.761773 21063 raft_consensus.cc:1261] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Refusing update from remote peer 5305461f474d43c798ff5cb998a9d8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:15.762424 21446 consensus_queue.cc:1035] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.762780 21424 consensus_queue.cc:1035] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:15.775940 20906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:15.776206 20906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:15.778785 20906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Sending DeleteTablet for 3 replicas of tablet 926151bf77eb471a8f658e07345f4b5c
I20260925 10:26:15.779565 21183 tablet_service.cc:1514] Processing DeleteTablet for tablet 926151bf77eb471a8f658e07345f4b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260925 10:26:15.779592 21321 tablet_service.cc:1514] Processing DeleteTablet for tablet 926151bf77eb471a8f658e07345f4b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46812
I20260925 10:26:15.779615 21042 tablet_service.cc:1514] Processing DeleteTablet for tablet 926151bf77eb471a8f658e07345f4b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:15 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:52760
I20260925 10:26:15.780762 20906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:15.781610 20906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:15.792465 21519 tablet_replica.cc:330] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: stopping tablet replica
I20260925 10:26:15.792557 21182 tablet_service.cc:1467] Processing CreateTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (DEFAULT_TABLE table=test-table [id=a939ec1603004d1f8107393b89c52c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.793004 21519 raft_consensus.cc:2229] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:15.793183 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.793646 21519 raft_consensus.cc:2258] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.793917 21322 tablet_service.cc:1467] Processing CreateTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (DEFAULT_TABLE table=test-table [id=a939ec1603004d1f8107393b89c52c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.794473 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.794903 21521 tablet_replica.cc:330] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: stopping tablet replica
I20260925 10:26:15.795039 21520 tablet_replica.cc:330] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: stopping tablet replica
I20260925 10:26:15.795130 21521 raft_consensus.cc:2229] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:15.795244 21520 raft_consensus.cc:2229] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:15.795403 21521 raft_consensus.cc:2258] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.795531 21520 raft_consensus.cc:2258] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:15.796494 21521 ts_tablet_manager.cc:1905] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.796620 21520 ts_tablet_manager.cc:1905] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.798374 21042 tablet_service.cc:1467] Processing CreateTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (DEFAULT_TABLE table=test-table [id=a939ec1603004d1f8107393b89c52c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:15.798795 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:15.801138 21501 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Bootstrap starting.
I20260925 10:26:15.802798 21501 tablet_bootstrap.cc:654] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.803233 21521 ts_tablet_manager.cc:1918] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:15.803401 21521 log.cc:1196] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/926151bf77eb471a8f658e07345f4b5c
I20260925 10:26:15.803876 21521 ts_tablet_manager.cc:1939] T 926151bf77eb471a8f658e07345f4b5c P 648aff6489e749c18f89af45bb156473: Deleting consensus metadata
I20260925 10:26:15.805053 21502 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Bootstrap starting.
I20260925 10:26:15.805013 20892 catalog_manager.cc:4841] TS 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449): tablet 926151bf77eb471a8f658e07345f4b5c (table test-table [id=b7c0e0d1a84740debfd9c76147645e35]) successfully deleted
I20260925 10:26:15.806401 21504 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Bootstrap starting.
I20260925 10:26:15.806700 21502 tablet_bootstrap.cc:654] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.807729 21504 tablet_bootstrap.cc:654] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:15.808859 21502 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: No bootstrap required, opened a new log
I20260925 10:26:15.809036 21502 ts_tablet_manager.cc:1397] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:26:15.809303 21501 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: No bootstrap required, opened a new log
I20260925 10:26:15.809484 21501 ts_tablet_manager.cc:1397] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260925 10:26:15.809878 21502 raft_consensus.cc:348] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.810120 21502 raft_consensus.cc:374] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.810196 21502 raft_consensus.cc:729] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4fd6bacb14071af4951d326998c99, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.810446 21502 consensus_queue.cc:260] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.810554 21501 raft_consensus.cc:348] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.810814 21501 raft_consensus.cc:374] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.811165 21520 ts_tablet_manager.cc:1918] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:15.811197 21501 raft_consensus.cc:729] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5305461f474d43c798ff5cb998a9d8c9, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.811299 21520 log.cc:1196] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/926151bf77eb471a8f658e07345f4b5c
I20260925 10:26:15.811627 21502 ts_tablet_manager.cc:1428] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:26:15.811774 21520 ts_tablet_manager.cc:1939] T 926151bf77eb471a8f658e07345f4b5c P 9fa4fd6bacb14071af4951d326998c99: Deleting consensus metadata
I20260925 10:26:15.811699 21501 consensus_queue.cc:260] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.812287 21501 ts_tablet_manager.cc:1428] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:15.812633 21519 ts_tablet_manager.cc:1905] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:15.813113 20894 catalog_manager.cc:4841] TS 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809): tablet 926151bf77eb471a8f658e07345f4b5c (table test-table [id=b7c0e0d1a84740debfd9c76147645e35]) successfully deleted
I20260925 10:26:15.815953 21504 tablet_bootstrap.cc:492] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: No bootstrap required, opened a new log
I20260925 10:26:15.816138 21504 ts_tablet_manager.cc:1397] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260925 10:26:15.816877 21504 raft_consensus.cc:348] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.817137 21504 raft_consensus.cc:374] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:15.817215 21504 raft_consensus.cc:729] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648aff6489e749c18f89af45bb156473, State: Initialized, Role: FOLLOWER
I20260925 10:26:15.817472 21504 consensus_queue.cc:260] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.818022 21504 ts_tablet_manager.cc:1428] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:26:15.818810 21519 ts_tablet_manager.cc:1918] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:15.818945 21519 log.cc:1196] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/926151bf77eb471a8f658e07345f4b5c
I20260925 10:26:15.819406 21519 ts_tablet_manager.cc:1939] T 926151bf77eb471a8f658e07345f4b5c P 5305461f474d43c798ff5cb998a9d8c9: Deleting consensus metadata
I20260925 10:26:15.820545 20893 catalog_manager.cc:4841] TS 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997): tablet 926151bf77eb471a8f658e07345f4b5c (table test-table [id=b7c0e0d1a84740debfd9c76147645e35]) successfully deleted
I20260925 10:26:15.959014 21483 raft_consensus.cc:482] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:15.959257 21483 raft_consensus.cc:504] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.959908 21483 leader_election.cc:288] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809), 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997)
I20260925 10:26:15.960570 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74bd7cb8b67408b8b6ab1d5cbc9c690" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99" is_pre_election: true
I20260925 10:26:15.960588 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74bd7cb8b67408b8b6ab1d5cbc9c690" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9" is_pre_election: true
I20260925 10:26:15.960837 21347 raft_consensus.cc:2454] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 0.
I20260925 10:26:15.960855 21207 raft_consensus.cc:2454] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 0.
I20260925 10:26:15.961321 20996 leader_election.cc:302] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5305461f474d43c798ff5cb998a9d8c9, 648aff6489e749c18f89af45bb156473, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:15.961681 21483 raft_consensus.cc:2781] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:15.961830 21483 raft_consensus.cc:482] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:15.961930 21483 raft_consensus.cc:3037] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.964030 21483 raft_consensus.cc:504] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.964638 21483 leader_election.cc:288] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 election: Requested vote from peers 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809), 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997)
I20260925 10:26:15.965103 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74bd7cb8b67408b8b6ab1d5cbc9c690" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa4fd6bacb14071af4951d326998c99"
I20260925 10:26:15.965178 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74bd7cb8b67408b8b6ab1d5cbc9c690" candidate_uuid: "648aff6489e749c18f89af45bb156473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5305461f474d43c798ff5cb998a9d8c9"
I20260925 10:26:15.965353 21207 raft_consensus.cc:3037] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.965402 21347 raft_consensus.cc:3037] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:15.967454 21347 raft_consensus.cc:2454] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 1.
I20260925 10:26:15.967475 21207 raft_consensus.cc:2454] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648aff6489e749c18f89af45bb156473 in term 1.
I20260925 10:26:15.967913 20997 leader_election.cc:302] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 648aff6489e749c18f89af45bb156473, 9fa4fd6bacb14071af4951d326998c99; no voters:
I20260925 10:26:15.968190 21483 raft_consensus.cc:2781] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:15.968559 21483 raft_consensus.cc:686] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 1 LEADER]: Becoming Leader. State: Replica: 648aff6489e749c18f89af45bb156473, State: Running, Role: LEADER
I20260925 10:26:15.968830 21483 consensus_queue.cc:237] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } }
I20260925 10:26:15.971470 20906 catalog_manager.cc:5495] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 reported cstate change: term changed from 0 to 1, leader changed from <none> to 648aff6489e749c18f89af45bb156473 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "648aff6489e749c18f89af45bb156473" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "648aff6489e749c18f89af45bb156473" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 } health_report { overall_health: UNKNOWN } } }
I20260925 10:26:16.020318 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.015475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.12.248.65:34449)
0925 10:26:16.015988 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:16.015992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:16.016011 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:26:16.016063 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:16.016066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:16.016125 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:16.016206 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:16.016615 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.017383 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.018744 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.019117 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.019211 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.019599 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:16.019704 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:16.019769 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:16.019896 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 10:26:16.019955 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
I20260925 10:26:16.033816 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.028660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:36933 (local address 127.12.248.65:34449)
0925 10:26:16.028858 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:16.028867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:16.029164 (+ 297us) server_negotiation.cc:407] Connection header received
0925 10:26:16.029368 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:16.029372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:16.029429 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:16.029555 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:16.029946 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.030702 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.032208 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.032581 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.032673 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.033119 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:16.033226 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:16.033309 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:16.033468 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:26:16.033557 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":51}
I20260925 10:26:16.033962 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.028599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34449 (local address 127.12.248.65:36933)
0925 10:26:16.029076 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:16.029092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:16.029267 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:16.029602 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:16.029606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:16.029616 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:26:16.029812 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:16.029820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.030876 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.030880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:16.032066 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:16.032078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.032738 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.032743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:16.032880 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.032937 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:26:16.033316 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:26:16.033325 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:26:16.033401 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:26:16.033604 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":18,"thread_start_us":257,"threads_started":1}
I20260925 10:26:16.041600 21347 raft_consensus.cc:1261] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Refusing update from remote peer 648aff6489e749c18f89af45bb156473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:16.041671 21207 raft_consensus.cc:1261] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Refusing update from remote peer 648aff6489e749c18f89af45bb156473: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:16.042203 21525 consensus_queue.cc:1035] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5305461f474d43c798ff5cb998a9d8c9" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:16.042521 21527 consensus_queue.cc:1035] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa4fd6bacb14071af4951d326998c99" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:16.103972 20906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:16.104254 20906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:16.107622 20906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5800b6b92a6d4b3fa788d5f933c332b0: Sending DeleteTablet for 3 replicas of tablet b74bd7cb8b67408b8b6ab1d5cbc9c690
I20260925 10:26:16.108408 21044 tablet_service.cc:1514] Processing DeleteTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:16 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:52760
I20260925 10:26:16.108626 21322 tablet_service.cc:1514] Processing DeleteTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:16 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46812
I20260925 10:26:16.108759 21182 tablet_service.cc:1514] Processing DeleteTablet for tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:16 UTC) from {username='oryx', principal='oryx/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:43958
I20260925 10:26:16.109061 21521 tablet_replica.cc:330] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: stopping tablet replica
I20260925 10:26:16.109172 21520 tablet_replica.cc:330] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: stopping tablet replica
I20260925 10:26:16.109284 21521 raft_consensus.cc:2229] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:16.109347 21520 raft_consensus.cc:2229] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:16.109549 21520 raft_consensus.cc:2258] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:16.109808 21519 tablet_replica.cc:330] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: stopping tablet replica
I20260925 10:26:16.109846 21521 raft_consensus.cc:2258] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:16.110008 21519 raft_consensus.cc:2229] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:16.110199 21519 raft_consensus.cc:2258] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:16.110795 21520 ts_tablet_manager.cc:1905] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:16.110975 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20978
I20260925 10:26:16.111680 21519 ts_tablet_manager.cc:1905] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:16.117282 21521 ts_tablet_manager.cc:1905] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:16.117702 21520 ts_tablet_manager.cc:1918] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:16.117874 21520 log.cc:1196] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/b74bd7cb8b67408b8b6ab1d5cbc9c690
I20260925 10:26:16.118287 21520 ts_tablet_manager.cc:1939] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 9fa4fd6bacb14071af4951d326998c99: Deleting consensus metadata
I20260925 10:26:16.118997 21519 ts_tablet_manager.cc:1918] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:16.119205 21519 log.cc:1196] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/b74bd7cb8b67408b8b6ab1d5cbc9c690
I20260925 10:26:16.119473 20894 catalog_manager.cc:4841] TS 9fa4fd6bacb14071af4951d326998c99 (127.12.248.66:37809): tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (table test-table [id=a939ec1603004d1f8107393b89c52c8f]) successfully deleted
I20260925 10:26:16.119750 21519 ts_tablet_manager.cc:1939] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 5305461f474d43c798ff5cb998a9d8c9: Deleting consensus metadata
I20260925 10:26:16.120751 20893 catalog_manager.cc:4841] TS 5305461f474d43c798ff5cb998a9d8c9 (127.12.248.67:35997): tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (table test-table [id=a939ec1603004d1f8107393b89c52c8f]) successfully deleted
I20260925 10:26:16.124505 21521 ts_tablet_manager.cc:1918] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:26:16.124682 21521 log.cc:1196] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/b74bd7cb8b67408b8b6ab1d5cbc9c690
I20260925 10:26:16.125134 21521 ts_tablet_manager.cc:1939] T b74bd7cb8b67408b8b6ab1d5cbc9c690 P 648aff6489e749c18f89af45bb156473: Deleting consensus metadata
I20260925 10:26:16.126245 20892 catalog_manager.cc:4841] TS 648aff6489e749c18f89af45bb156473 (127.12.248.65:34449): tablet b74bd7cb8b67408b8b6ab1d5cbc9c690 (table test-table [id=a939ec1603004d1f8107393b89c52c8f]) successfully deleted
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.65@KRBTEST.COM
I20260925 10:26:16.140462 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.113115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.12.248.65:34449)
0925 10:26:16.113318 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:16.113326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:16.121088 (+ 7762us) server_negotiation.cc:407] Connection header received
0925 10:26:16.121305 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:16.121318 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:16.121400 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:16.121566 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:16.123109 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.124536 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.126727 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.127401 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.127550 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.132791 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:16.132816 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:16.132837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:16.132882 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:16.137031 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.137963 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.137973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.137977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.138152 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.138588 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.138593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.138597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.139028 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:16.139409 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:16.139616 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:26:16.139842 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:26:16.141167 21109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:16.271353 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20978
I20260925 10:26:16.294598 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21118
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.66@KRBTEST.COM
I20260925 10:26:16.309180 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.295384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.12.248.66:37809)
0925 10:26:16.295948 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:16.295953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:16.295969 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:26:16.296023 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:16.296026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:16.296084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:16.296175 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:16.297387 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.298177 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.299361 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.299797 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.299891 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.303955 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:16.303969 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:16.303985 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:16.304017 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:16.306908 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.307519 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.307525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.307528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.307655 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.307961 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.307964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.307966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.308306 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:16.308535 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:16.308708 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:26:16.308895 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":293,"threads_started":1}
I20260925 10:26:16.309921 21248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:16.438148 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21118
I20260925 10:26:16.472419 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21257
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[20862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.248.67@KRBTEST.COM
I20260925 10:26:16.490409 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:16.473364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.12.248.67:35997)
0925 10:26:16.473892 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:16.473898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:16.473915 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:16.473976 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:16.473981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:16.474060 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:16.474169 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:16.476231 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.477324 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.479454 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:16.480148 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:16.480269 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:16.485186 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:16.485201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:16.485217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:16.485260 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:16.487995 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.488608 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.488614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.488617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.488734 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:16.489030 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:16.489033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:16.489035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:16.489359 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:16.489630 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:16.489901 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:26:16.490134 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":273,"threads_started":1}
I20260925 10:26:16.491196 21387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:16.613723 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21257
I20260925 10:26:16.635311 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 20875
I20260925 10:26:16.636453 20947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:16.754429 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 20875
2026-09-25T10:26:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6134 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[21546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[21546](info): set up 2 sockets
Sep 25 10:26:16 dist-test-slave-f9rl krb5kdc[21546](info): commencing operation
krb5kdc: starting...
W20260925 10:26:17.919807 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:26:18 dist-test-slave-f9rl krb5kdc[21546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:26:18.145117 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:41783
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:18.474603 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:18.474905 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:18.475071 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:18.482376 21559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:18.482491 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:18.482592 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:18.482666 21559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:18.482759 21559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:18.492682 21559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:18.494066 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:18.495484 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:18.507108 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:18.507236 21567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:18.507160 21565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:18.508322 21559 server_base.cc:1034] running on GCE node
I20260925 10:26:18.509125 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:18.510619 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:18.511849 21559 hybrid_clock.cc:648] HybridClock initialized: now 1790331978511799 us; error 33 us; skew 500 ppm
I20260925 10:26:18.512217 21559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:26:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1827 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:26:18 dist-test-slave-f9rl krb5kdc[21573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:26:18 dist-test-slave-f9rl krb5kdc[21573](info): set up 2 sockets
Sep 25 10:26:18 dist-test-slave-f9rl krb5kdc[21573](info): commencing operation
krb5kdc: starting...
W20260925 10:26:19.751857 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:26:19 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:26:19.982062 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43993
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:20.309623 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:20.309966 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:20.310127 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:20.317404 21586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:20.317579 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:20.317688 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:20.317790 21586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:20.317878 21586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:20.327823 21586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:20.329229 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:20.330626 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:20.342684 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:20.342732 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:20.342765 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:20.343727 21586 server_base.cc:1034] running on GCE node
I20260925 10:26:20.344602 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:20.346127 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:20.347363 21586 hybrid_clock.cc:648] HybridClock initialized: now 1790331980347318 us; error 53 us; skew 500 ppm
I20260925 10:26:20.347713 21586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:20 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:20.358247 21586 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:26:20.359782 21586 webserver.cc:460] Webserver started at http://127.12.248.126:38645/ using document root <none> and password file <none>
I20260925 10:26:20.360489 21586 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:20.360590 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:20.360870 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:20.363996 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "2adc4403f04442e682f4e2cea0bbd27e"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "ad5cc2a534c96be11eafbdc2d4e44a8d"
server_key_iv: "4145a057631062557b1377a44ddd279e"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.364744 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "2adc4403f04442e682f4e2cea0bbd27e"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "ad5cc2a534c96be11eafbdc2d4e44a8d"
server_key_iv: "4145a057631062557b1377a44ddd279e"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.370064 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:26:20.373492 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:20.374800 21586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:26:20.374938 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "2adc4403f04442e682f4e2cea0bbd27e"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "ad5cc2a534c96be11eafbdc2d4e44a8d"
server_key_iv: "4145a057631062557b1377a44ddd279e"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.375093 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:20.412400 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:20.416503 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:20.416749 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:20.436153 21586 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:43993
I20260925 10:26:20.436177 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:43993 every 8 connection(s)
I20260925 10:26:20.437641 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:20.441357 21653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:20.443579 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21586
I20260925 10:26:20.443813 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:20.444633 13281 external_mini_cluster.cc:1428] Setting key 8776e88f1ee341cb348597e8fece60a7
I20260925 10:26:20.453337 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Bootstrap starting.
I20260925 10:26:20.456579 21653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:20.457669 21653 log.cc:824] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:26:20 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:20.460698 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: No bootstrap required, opened a new log
I20260925 10:26:20.467084 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:20.446867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.12.248.126:43993)
0925 10:26:20.447629 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:20.447641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:20.447683 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:26:20.449379 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:20.449401 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:20.449853 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:20.450232 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:20.451715 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:20.453009 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:20.454362 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:20.455056 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:20.455273 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:20.459607 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:20.459614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:20.459628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:20.459666 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:20.462832 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:20.463679 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:20.463685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:20.463690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:20.463885 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:20.464281 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:20.464284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:20.464286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:20.464828 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:20.465035 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:20.465541 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 10:26:20.465990 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":345,"threads_started":1}
I20260925 10:26:20.470067 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:20.470474 21653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:20.470575 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adc4403f04442e682f4e2cea0bbd27e, State: Initialized, Role: FOLLOWER
I20260925 10:26:20.471110 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:20.471364 21653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:20.471503 21653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:20.471665 21653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:20.473933 21653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:20.474440 21653 leader_election.cc:302] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2adc4403f04442e682f4e2cea0bbd27e; no voters:
I20260925 10:26:20.475001 21653 leader_election.cc:288] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:20.475140 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:20.477322 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 LEADER]: Becoming Leader. State: Replica: 2adc4403f04442e682f4e2cea0bbd27e, State: Running, Role: LEADER
I20260925 10:26:20.477818 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:20.478001 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:20.481014 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2adc4403f04442e682f4e2cea0bbd27e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } } }
I20260925 10:26:20.481520 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:20.481416 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adc4403f04442e682f4e2cea0bbd27e. Latest consensus state: current_term: 1 leader_uuid: "2adc4403f04442e682f4e2cea0bbd27e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } } }
I20260925 10:26:20.481741 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:20.482685 21667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:20.487617 21667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:20.497260 21667 catalog_manager.cc:1347] Generated new cluster ID: 821ba17d0a9e48fc9b30fa90176f68ec
I20260925 10:26:20.497390 21667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:20.510005 21667 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:20.511914 21667 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:20.523970 21667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Generated new TSK 0
I20260925 10:26:20.525146 21667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:26:20.584623 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:20.924191 21679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:20.924468 21679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:20.924618 21679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:20.932058 21679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:20.932209 21679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:20.932385 21679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:20.942744 21679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:20.944128 21679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:20.945600 21679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:20.959714 21687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:20.959764 21684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:20.959792 21685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:20.960749 21679 server_base.cc:1034] running on GCE node
I20260925 10:26:20.961341 21679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:20.962386 21679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:20.963610 21679 hybrid_clock.cc:648] HybridClock initialized: now 1790331980963542 us; error 56 us; skew 500 ppm
I20260925 10:26:20.963986 21679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:20 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:20.967948 21679 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:26:20.969621 21679 webserver.cc:460] Webserver started at http://127.12.248.65:39561/ using document root <none> and password file <none>
I20260925 10:26:20.970353 21679 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:20.970496 21679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:20.970831 21679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:20.973883 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "86a3c4d7c6d849e0b61ca40c3b98fb45"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "69b5c554d427fd538e0484d7152fee9c"
server_key_iv: "b68373b2016ae9df832bf439ea8c0372"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.974692 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "86a3c4d7c6d849e0b61ca40c3b98fb45"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "69b5c554d427fd538e0484d7152fee9c"
server_key_iv: "b68373b2016ae9df832bf439ea8c0372"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.980206 21679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260925 10:26:20.983659 21694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:20.984961 21679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:20.985150 21679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "86a3c4d7c6d849e0b61ca40c3b98fb45"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "69b5c554d427fd538e0484d7152fee9c"
server_key_iv: "b68373b2016ae9df832bf439ea8c0372"
server_key_version: "encryptionkey@0"
I20260925 10:26:20.985360 21679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:21.004935 21679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:21.008883 21679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:21.009192 21679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:21.010210 21679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:21.011586 21679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:21.011693 21679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:21.011842 21679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:21.011910 21679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:21.045130 21679 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:34601
I20260925 10:26:21.045177 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:34601 every 8 connection(s)
I20260925 10:26:21.046521 21679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:21.047434 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21679
I20260925 10:26:21.047644 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:21.048463 13281 external_mini_cluster.cc:1428] Setting key 439fef7efe0dd779a42eaefd3f05c4b6
Sep 25 10:26:21 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331980, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:21.069669 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:21.049531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:39285 (local address 127.12.248.126:43993)
0925 10:26:21.050026 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:21.050031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:21.051172 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 10:26:21.053048 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:21.053054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:21.053143 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:21.053257 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:21.054879 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.055609 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.057679 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.058081 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.058194 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:21.062420 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:21.062429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:21.062437 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:21.062482 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:21.066050 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:21.066863 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:21.066872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:21.066878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:21.067039 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:21.067486 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:21.067494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:21.067500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:21.067911 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:21.068094 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:21.068834 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 10:26:21.069127 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
I20260925 10:26:21.070294 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:21.050232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:39285)
0925 10:26:21.051019 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:21.051062 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:21.052768 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:21.053572 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:21.053582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:21.054125 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:21.054707 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:21.054724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.055774 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.055780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:21.057517 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:21.057534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.058329 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.058339 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:21.058679 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:21.059435 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:21.059475 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:21.062169 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:21.066208 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:21.066215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:21.066231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:21.066714 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:21.067151 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:21.067154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:21.067156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:21.067359 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:21.068035 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:21.068041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:21.068412 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 10:26:21.068650 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 10:26:21.069058 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":317,"threads_started":1}
I20260925 10:26:21.072149 21807 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43993
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:26:21.072497 21807 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:21.073258 21807 heartbeater.cc:511] Master 127.12.248.126:43993 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
I20260925 10:26:21.075572 21618 ts_manager.cc:194] Registered new tserver with Master: 86a3c4d7c6d849e0b61ca40c3b98fb45 (127.12.248.65:34601)
I20260925 10:26:21.077016 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:39285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:26:21.105654 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:21.432674 21813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:21.432976 21813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:21.433156 21813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:21.443813 21813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:21.443964 21813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:21.444149 21813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:21.459685 21813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:21.460976 21813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:21.462329 21813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:21.475581 21819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:21.475529 21821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:21.475626 21818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:21.476651 21813 server_base.cc:1034] running on GCE node
I20260925 10:26:21.477279 21813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:21.478394 21813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:21.479635 21813 hybrid_clock.cc:648] HybridClock initialized: now 1790331981479562 us; error 50 us; skew 500 ppm
I20260925 10:26:21.479988 21813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:21 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:21.483171 21813 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:26:21.484599 21813 webserver.cc:460] Webserver started at http://127.12.248.66:36117/ using document root <none> and password file <none>
I20260925 10:26:21.485301 21813 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:21.485510 21813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:21.485888 21813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:21.488981 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "4dcbe506fbab43508dda42f4d64026a5"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "08d1a22088f010812362715cfe5250e1"
server_key_iv: "b41d67fc0576af20afde68217cffa6ac"
server_key_version: "encryptionkey@0"
I20260925 10:26:21.489882 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "4dcbe506fbab43508dda42f4d64026a5"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "08d1a22088f010812362715cfe5250e1"
server_key_iv: "b41d67fc0576af20afde68217cffa6ac"
server_key_version: "encryptionkey@0"
I20260925 10:26:21.494897 21813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:26:21.498490 21828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:21.499809 21813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:26:21.499970 21813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "4dcbe506fbab43508dda42f4d64026a5"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "08d1a22088f010812362715cfe5250e1"
server_key_iv: "b41d67fc0576af20afde68217cffa6ac"
server_key_version: "encryptionkey@0"
I20260925 10:26:21.500164 21813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:21.517050 21813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:21.520979 21813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:21.521277 21813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:21.522311 21813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:21.523620 21813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:21.523720 21813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:21.523865 21813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:21.523934 21813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:21.554836 21813 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:43531
I20260925 10:26:21.554873 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:43531 every 8 connection(s)
I20260925 10:26:21.556118 21813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:21.557866 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21813
I20260925 10:26:21.558065 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:21.558584 13281 external_mini_cluster.cc:1428] Setting key 22fb880aa2da3aab09485b76d4787acb
Sep 25 10:26:21 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:21.579430 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:21.559084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:37353 (local address 127.12.248.126:43993)
0925 10:26:21.559639 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:21.559646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:21.560764 (+ 1118us) server_negotiation.cc:407] Connection header received
0925 10:26:21.563686 (+ 2922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:21.563690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:21.563754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:21.563851 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:21.566000 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.567099 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.569319 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.569816 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.569952 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:21.573701 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:21.573705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:21.573708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:21.573742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:21.576206 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:21.576934 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:21.576943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:21.576947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:21.577063 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:21.577532 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:21.577536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:21.577538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:21.577792 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:21.577963 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:21.578792 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 10:26:21.579023 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260925 10:26:21.580420 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:21.559748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:37353)
0925 10:26:21.560591 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:21.560646 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:21.563388 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:21.564121 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:21.564135 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:21.564913 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:21.565775 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:21.565805 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.567278 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.567286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:21.569170 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:21.569181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:21.570047 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:21.570054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:21.570349 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:21.571086 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:21.571118 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:21.573424 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:21.576364 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:21.576372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:21.576389 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:21.576809 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:21.577172 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:21.577176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:21.577180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:21.577367 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:21.577963 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:21.577971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:21.578424 (+ 453us) client_negotiation.cc:769] Sending connection context
0925 10:26:21.578702 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 10:26:21.579246 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":299,"threads_started":1}
I20260925 10:26:21.582109 21941 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43993
I20260925 10:26:21.582419 21941 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:26:21.583135 21941 heartbeater.cc:511] Master 127.12.248.126:43993 requested a full tablet report, sending...
I20260925 10:26:21.584614 21618 ts_manager.cc:194] Registered new tserver with Master: 4dcbe506fbab43508dda42f4d64026a5 (127.12.248.66:43531)
I20260925 10:26:21.585497 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:37353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:26:21.617053 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:21.952781 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:21.953112 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:21.953287 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:21.960580 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:21.960736 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:21.960912 21946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:21.971100 21946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:21.972446 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:21.973871 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:21.987836 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:21.987862 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:21.987920 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:21.988914 21946 server_base.cc:1034] running on GCE node
I20260925 10:26:21.989619 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:21.990619 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:21.991796 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790331981991733 us; error 55 us; skew 500 ppm
I20260925 10:26:21.992168 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:21 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:21.995342 21946 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:26:21.996812 21946 webserver.cc:460] Webserver started at http://127.12.248.67:46049/ using document root <none> and password file <none>
I20260925 10:26:21.997519 21946 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:21.997650 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:21.998008 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:22.001030 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "0739e1904c88468f9f43ea0e58339811"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "4b7e783081adc77ef7770708ee2a38f3"
server_key_iv: "9604936a1180ff1a839fec6366914a9f"
server_key_version: "encryptionkey@0"
I20260925 10:26:22.001899 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "0739e1904c88468f9f43ea0e58339811"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "4b7e783081adc77ef7770708ee2a38f3"
server_key_iv: "9604936a1180ff1a839fec6366914a9f"
server_key_version: "encryptionkey@0"
I20260925 10:26:22.007157 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 10:26:22.010696 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:22.012086 21946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:26:22.012261 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0739e1904c88468f9f43ea0e58339811"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "4b7e783081adc77ef7770708ee2a38f3"
server_key_iv: "9604936a1180ff1a839fec6366914a9f"
server_key_version: "encryptionkey@0"
I20260925 10:26:22.012466 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:22.026242 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:22.030779 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:22.031095 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:22.032119 21946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:22.033622 21946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:22.033712 21946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:22.033896 21946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:22.033978 21946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:22.069624 21946 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:39233
I20260925 10:26:22.069664 22073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:39233 every 8 connection(s)
I20260925 10:26:22.070933 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:22.079579 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 21946
I20260925 10:26:22.079761 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:22.080339 13281 external_mini_cluster.cc:1428] Setting key 6154521aab87ed54dd5d2d22c40012d9
I20260925 10:26:22.080335 21807 heartbeater.cc:503] Master 127.12.248.126:43993 was elected leader, sending a full tablet report...
Sep 25 10:26:22 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:22.092114 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:22.073625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:46137 (local address 127.12.248.126:43993)
0925 10:26:22.073804 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:22.073808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:22.074965 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 10:26:22.076816 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:22.076822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:22.076924 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:22.077034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:22.078665 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.079590 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.082349 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.082847 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.082945 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:22.086631 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:22.086636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:22.086639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:22.086670 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:22.089536 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.090185 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.090189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.090191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.090340 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.090717 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.090720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.090722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.090970 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:22.091078 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:22.091699 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 10:26:22.091857 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:26:22.093057 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:22.074111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:46137)
0925 10:26:22.074808 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:22.074844 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:22.076537 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:22.077258 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:22.077267 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:22.077874 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:22.078487 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:22.078506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.079804 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.079809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:22.082195 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:22.082211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.083069 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.083078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:22.083374 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:22.084069 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:22.084091 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:22.086434 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:22.089664 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:22.089670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:22.089682 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:22.090077 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:22.090435 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:22.090438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:22.090441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:22.090615 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:22.091072 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:22.091078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:22.091388 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 10:26:22.091603 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 10:26:22.091980 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":270,"threads_started":1}
I20260925 10:26:22.094726 22074 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43993
I20260925 10:26:22.095095 22074 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:22.096005 22074 heartbeater.cc:511] Master 127.12.248.126:43993 requested a full tablet report, sending...
I20260925 10:26:22.097354 21618 ts_manager.cc:194] Registered new tserver with Master: 0739e1904c88468f9f43ea0e58339811 (127.12.248.67:39233)
I20260925 10:26:22.098423 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:46137
I20260925 10:26:22.108044 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
Entry for principal oryx/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab.
WARNING: no policy specified for oryx/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
Entry for principal oryx/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab.
WARNING: no policy specified for oryx/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
Entry for principal oryx/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab.
I20260925 10:26:22.244390 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21679
Sep 25 10:26:22 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:26:22.259999 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:22.245607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.12.248.65:34601)
0925 10:26:22.246309 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:22.246316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:22.246346 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:26:22.246409 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:22.246416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:22.246637 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:22.246810 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:22.248106 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.249150 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.250353 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.250925 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.251020 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:22.254936 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:22.254947 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:22.254961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:22.254993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:22.257690 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.258291 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.258298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.258300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.258414 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.258712 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.258715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.258717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.259042 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:22.259263 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:22.259502 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:26:22.259691 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":381,"threads_started":1}
I20260925 10:26:22.261314 21802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:22.346725 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21679
I20260925 10:26:22.364780 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21813
Sep 25 10:26:22 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:26:22.380714 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:22.365697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.12.248.66:43531)
0925 10:26:22.366206 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:22.366214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:22.366237 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:26:22.366296 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:22.366302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:22.366508 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:22.366676 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:22.367918 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.368905 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.370284 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.370829 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.370970 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:22.375161 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:22.375172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:22.375185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:22.375217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:22.377987 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.378752 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.378777 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.378779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.378895 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.379299 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.379303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.379305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.379674 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:22.379939 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:22.380130 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:26:22.380330 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260925 10:26:22.382035 21936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:22.468979 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21813
I20260925 10:26:22.485394 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21946
Sep 25 10:26:22 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:26:22.502449 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:22.486596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.12.248.67:39233)
0925 10:26:22.487281 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:22.487288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:22.487317 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:26:22.487375 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:22.487381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:22.487593 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:22.487763 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:22.489128 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.490237 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.491556 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:22.492164 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:22.492313 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:22.496736 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:22.496747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:22.496761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:22.496800 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:22.499606 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.500289 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.500303 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.500305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.500425 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:22.500818 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:22.500824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:22.500831 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:22.501156 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:22.501512 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:22.501822 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 10:26:22.502089 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":400,"threads_started":1}
I20260925 10:26:22.503844 22069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:22.586144 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21946
I20260925 10:26:22.602130 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 21586
I20260925 10:26:22.603365 21648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:22.706215 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 21586
I20260925 10:26:22.727183 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43993
--webserver_interface=127.12.248.126
--webserver_port=38645
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:23.056318 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:23.056622 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:23.056787 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:23.063973 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:23.064080 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:23.064174 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:23.064282 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:23.064366 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:23.074398 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=38645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:23.075743 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:23.077081 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:23.089288 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:23.089355 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:23.089504 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:23.093238 22091 server_base.cc:1034] running on GCE node
I20260925 10:26:23.094244 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:23.096143 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:23.097450 22091 hybrid_clock.cc:648] HybridClock initialized: now 1790331983097351 us; error 70 us; skew 500 ppm
I20260925 10:26:23.097815 22091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331983, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:23.100802 22091 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:26:23.102541 22091 webserver.cc:460] Webserver started at http://127.12.248.126:38645/ using document root <none> and password file <none>
I20260925 10:26:23.103504 22091 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:23.103663 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:23.109572 22091 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:26:23.112432 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:23.114108 22091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:23.114331 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "2adc4403f04442e682f4e2cea0bbd27e"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "ad5cc2a534c96be11eafbdc2d4e44a8d"
server_key_iv: "4145a057631062557b1377a44ddd279e"
server_key_version: "encryptionkey@0"
I20260925 10:26:23.115368 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:23.150079 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:23.154083 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:23.154376 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:23.172920 22091 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:43993
I20260925 10:26:23.172932 22157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:43993 every 8 connection(s)
I20260925 10:26:23.174629 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:23.180377 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22091
I20260925 10:26:23.181401 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:34601
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:43993
--webserver_port=39561
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:26:23.182328 22158 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:26:23.187579 22158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Bootstrap starting.
I20260925 10:26:23.207485 22158 log.cc:824] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:23.216436 22158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:26:23.216979 22158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Bootstrap complete.
I20260925 10:26:23.226480 22158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:23.227414 22158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adc4403f04442e682f4e2cea0bbd27e, State: Initialized, Role: FOLLOWER
I20260925 10:26:23.228021 22158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:23.228281 22158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:23.228411 22158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:23.228603 22158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:26:23.232481 22158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:23.232999 22158 leader_election.cc:302] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2adc4403f04442e682f4e2cea0bbd27e; no voters:
I20260925 10:26:23.233685 22158 leader_election.cc:288] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:26:23.233829 22162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:26:23.235090 22162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [term 2 LEADER]: Becoming Leader. State: Replica: 2adc4403f04442e682f4e2cea0bbd27e, State: Running, Role: LEADER
I20260925 10:26:23.235682 22162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } }
I20260925 10:26:23.236106 22158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:23.238174 22164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adc4403f04442e682f4e2cea0bbd27e. Latest consensus state: current_term: 2 leader_uuid: "2adc4403f04442e682f4e2cea0bbd27e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } } }
I20260925 10:26:23.238543 22164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:23.238445 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2adc4403f04442e682f4e2cea0bbd27e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adc4403f04442e682f4e2cea0bbd27e" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43993 } } }
I20260925 10:26:23.239320 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:23.240381 22169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:23.247098 22169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:23.248729 22169 catalog_manager.cc:1259] Loaded cluster ID: 821ba17d0a9e48fc9b30fa90176f68ec
I20260925 10:26:23.248898 22169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:23.252947 22169 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:23.254585 22169 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2adc4403f04442e682f4e2cea0bbd27e: Loaded TSK: 0
I20260925 10:26:23.256194 22169 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:26:23.528389 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:23.528734 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:23.528891 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:23.536556 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:23.536707 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:23.536897 22160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:23.547636 22160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.65.keytab
--principal=oryx/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:34601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=39561
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:23.549127 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:23.550652 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:23.565222 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:23.565488 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:23.566295 22160 server_base.cc:1034] running on GCE node
W20260925 10:26:23.566551 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:23.567203 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:23.568214 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:23.569490 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790331983569399 us; error 49 us; skew 500 ppm
I20260925 10:26:23.569826 22160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331983, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:23.573021 22160 init.cc:377] Logged in from keytab as oryx/127.12.248.65@KRBTEST.COM (short username oryx)
I20260925 10:26:23.574676 22160 webserver.cc:460] Webserver started at http://127.12.248.65:39561/ using document root <none> and password file <none>
I20260925 10:26:23.575551 22160 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:23.575700 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:23.581315 22160 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:26:23.584285 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:23.585836 22160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:23.586024 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "86a3c4d7c6d849e0b61ca40c3b98fb45"
format_stamp: "Formatted at 2026-09-25 10:26:20 on dist-test-slave-f9rl"
server_key: "69b5c554d427fd538e0484d7152fee9c"
server_key_iv: "b68373b2016ae9df832bf439ea8c0372"
server_key_version: "encryptionkey@0"
I20260925 10:26:23.587040 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:23.621726 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:23.625759 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:23.626075 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:23.627112 22160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:23.628440 22160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:23.628566 22160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:23.628713 22160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:23.628803 22160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:23.662174 22160 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:34601
I20260925 10:26:23.662231 22306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:34601 every 8 connection(s)
I20260925 10:26:23.663698 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:23.668131 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22160
I20260925 10:26:23.668771 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:43531
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:43993
--webserver_port=36117
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:23.684218 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:23.667052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:56733)
0925 10:26:23.667869 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:23.667916 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:23.669804 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:23.671471 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:23.671480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:23.671986 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:23.672536 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.672552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.674948 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.674953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.676797 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.676814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.677909 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.677919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.678269 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.679004 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:23.679039 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:23.683089 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":364,"threads_started":1}
W20260925 10:26:23.684737 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:23.666938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:56733 (local address 127.12.248.126:43993)
0925 10:26:23.667649 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:23.667667 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:23.668067 (+ 400us) server_negotiation.cc:407] Connection header received
0925 10:26:23.670440 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:23.670463 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:23.670935 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:23.671345 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:23.672742 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.674786 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.676955 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.677676 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.677820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.683545 (+ 5725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:56733: BlockingRecv error: recv got EOF from 127.12.248.65:56733 (error 108)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":276,"threads_started":1}
W20260925 10:26:23.684968 22307 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.248.126:43993: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:26:23.694818 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:23.685825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:47321 (local address 127.12.248.126:43993)
0925 10:26:23.686023 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:23.686031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:23.686052 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:26:23.686110 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:23.686118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:23.686200 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:23.686307 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:23.686926 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.688264 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.689421 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.690257 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.690409 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.694659 (+ 4250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:47321: BlockingRecv error: recv got EOF from 127.12.248.65:47321 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:26:23.694828 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:23.685648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:47321)
0925 10:26:23.685865 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:23.685885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:23.685989 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:23.686334 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:23.686337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:23.686617 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:23.686751 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.686758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.688404 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.688408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.689280 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.689290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.690397 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.690401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.690571 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.691218 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:23.691250 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:23.694502 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:23 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:23.703472 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:23.695674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:40005)
0925 10:26:23.695845 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:23.695865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:23.695959 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:23.696282 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:23.696286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:23.696581 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:23.696715 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.696731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.697873 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.697878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.698798 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:23.698809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.699689 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.699694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:23.699868 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.700445 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:23.700461 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:23.703146 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 10:26:23.703637 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:23.695767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:40005 (local address 127.12.248.126:43993)
0925 10:26:23.695945 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:23.695951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:23.695968 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:23.696059 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:23.696064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:23.696144 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:23.696262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:23.696856 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.697741 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.698949 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:23.699539 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:23.699665 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:23.703364 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:40005: BlockingRecv error: recv got EOF from 127.12.248.65:40005 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 10:26:23.703860 22307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:26:24.003585 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:24.003912 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:24.004084 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:24.011183 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:24.011350 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:24.011530 22311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:24.021768 22311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.66.keytab
--principal=oryx/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:43531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=36117
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:24.023178 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:24.024518 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:24.037984 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:24.038007 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:24.037981 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:24.038925 22311 server_base.cc:1034] running on GCE node
I20260925 10:26:24.039702 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:24.040673 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:24.041878 22311 hybrid_clock.cc:648] HybridClock initialized: now 1790331984041834 us; error 33 us; skew 500 ppm
I20260925 10:26:24.042230 22311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331984, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:24.045680 22311 init.cc:377] Logged in from keytab as oryx/127.12.248.66@KRBTEST.COM (short username oryx)
I20260925 10:26:24.047241 22311 webserver.cc:460] Webserver started at http://127.12.248.66:36117/ using document root <none> and password file <none>
I20260925 10:26:24.047935 22311 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:24.048051 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:24.054697 22311 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 10:26:24.057613 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.059098 22311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:26:24.059320 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "4dcbe506fbab43508dda42f4d64026a5"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "08d1a22088f010812362715cfe5250e1"
server_key_iv: "b41d67fc0576af20afde68217cffa6ac"
server_key_version: "encryptionkey@0"
I20260925 10:26:24.060360 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:24.082945 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:24.088341 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:24.088698 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:24.089802 22311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:24.091321 22311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:24.091460 22311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.091635 22311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:24.091749 22311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.128888 22311 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:43531
I20260925 10:26:24.128978 22438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:43531 every 8 connection(s)
I20260925 10:26:24.130384 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:24.135514 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22311
I20260925 10:26:24.136191 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:39233
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:43993
--webserver_port=46049
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:26:24.146903 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.132923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:57577 (local address 127.12.248.126:43993)
0925 10:26:24.133088 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.133093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.134368 (+ 1275us) server_negotiation.cc:407] Connection header received
0925 10:26:24.136356 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.136361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.136454 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.136566 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.138428 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.139153 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.140784 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.141184 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.141326 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.146657 (+ 5331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:57577: BlockingRecv error: recv got EOF from 127.12.248.66:57577 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:26:24.147830 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.133408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:57577)
0925 10:26:24.134196 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.134234 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.136067 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.136915 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.136923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.137663 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.138254 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.138272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.139325 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.139331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.140635 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.140647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.141458 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.141466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.141699 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.142482 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.142508 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.146461 (+ 3953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":275,"threads_started":1}
W20260925 10:26:24.148463 22439 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.248.126:43993: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:24.156863 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.149073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:35633)
0925 10:26:24.149261 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.149281 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.149384 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.149735 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.149740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.150063 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.150202 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.150209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.151168 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.151174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.152039 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.152049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.152768 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.152773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.152943 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.153591 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.153607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.156533 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:26:24.156980 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.149182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:35633 (local address 127.12.248.126:43993)
0925 10:26:24.149365 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.149370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.149387 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:24.149539 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.149544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.149607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.149711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.150344 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.151029 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.152182 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.152610 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.152744 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.156736 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:35633: BlockingRecv error: recv got EOF from 127.12.248.66:35633 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:24.166002 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.157897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:49515)
0925 10:26:24.158083 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.158102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.158202 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.158531 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.158536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.158856 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.158993 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.159023 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.159992 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.159997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.160897 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.160908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.161650 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.161656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.161864 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.162459 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.162477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.165661 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 10:26:24.166134 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.157998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:49515 (local address 127.12.248.126:43993)
0925 10:26:24.158172 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.158177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.158194 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:26:24.158345 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.158349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.158407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.158507 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.159163 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.159858 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.161035 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.161502 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.161625 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.165905 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:49515: BlockingRecv error: recv got EOF from 127.12.248.66:49515 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:26:24.166353 22439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:26:24.465957 22442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:24.466300 22442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:24.466457 22442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:24.473641 22442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:24.473784 22442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:24.473974 22442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:24.483662 22442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/krb5kdc/oryx_127.12.248.67.keytab
--principal=oryx/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:39233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=46049
--tserver_master_addrs=127.12.248.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:24.485035 22442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:24.486415 22442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:24.499377 22447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:24.499521 22448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:24.499497 22450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:24.500066 22442 server_base.cc:1034] running on GCE node
I20260925 10:26:24.500686 22442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:24.501787 22442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:24.503010 22442 hybrid_clock.cc:648] HybridClock initialized: now 1790331984502958 us; error 49 us; skew 500 ppm
I20260925 10:26:24.503322 22442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331984, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:24.506489 22442 init.cc:377] Logged in from keytab as oryx/127.12.248.67@KRBTEST.COM (short username oryx)
I20260925 10:26:24.507928 22442 webserver.cc:460] Webserver started at http://127.12.248.67:46049/ using document root <none> and password file <none>
I20260925 10:26:24.508620 22442 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:24.508775 22442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:24.514114 22442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:26:24.516700 22457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.518100 22442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:26:24.518286 22442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0739e1904c88468f9f43ea0e58339811"
format_stamp: "Formatted at 2026-09-25 10:26:21 on dist-test-slave-f9rl"
server_key: "4b7e783081adc77ef7770708ee2a38f3"
server_key_iv: "9604936a1180ff1a839fec6366914a9f"
server_key_version: "encryptionkey@0"
I20260925 10:26:24.519232 22442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:24.538938 22442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:24.542985 22442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:24.543275 22442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:24.544361 22442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:24.545707 22442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:24.545850 22442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.545989 22442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:24.546085 22442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:24.579276 22442 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:39233
I20260925 10:26:24.579360 22569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:39233 every 8 connection(s)
I20260925 10:26:24.580740 22442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:24.590107 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22442
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:26:24.597725 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.583330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:60781 (local address 127.12.248.126:43993)
0925 10:26:24.583510 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.583514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.584548 (+ 1034us) server_negotiation.cc:407] Connection header received
0925 10:26:24.586651 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.586656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.586749 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.586871 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.588351 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.589053 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.591411 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.592427 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.592597 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.597413 (+ 4816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:60781: BlockingRecv error: recv got EOF from 127.12.248.67:60781 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:26:24.598501 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.583709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:60781)
0925 10:26:24.584380 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.584426 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.586359 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.587112 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.587120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.587606 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.588170 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.588186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.589214 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.589219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.591220 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.591235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.592664 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.592669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.592879 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.593594 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.593621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.597157 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
W20260925 10:26:24.599243 22570 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.248.126:43993: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:24.608556 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.599853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:38761)
0925 10:26:24.600047 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.600072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.600148 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.600431 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.600434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.600703 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.600831 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.600838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.602108 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.602122 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.603268 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.603287 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.604501 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.604509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.604655 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.605146 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.605162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.608292 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 10:26:24.609077 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.599974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38761 (local address 127.12.248.126:43993)
0925 10:26:24.600144 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.600150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.600170 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:26:24.600222 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.600229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.600301 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.600405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.600965 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.601980 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.603422 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.604319 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.604484 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.608540 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:38761: BlockingRecv error: recv got EOF from 127.12.248.67:38761 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:26:24.610265 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.591296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.12.248.126:43993)
0925 10:26:24.591825 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.591829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.593294 (+ 1465us) server_negotiation.cc:407] Connection header received
0925 10:26:24.593477 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.593482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.593551 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.593660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.595420 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.596256 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.597836 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.598459 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.598598 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.601965 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:24.601975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:24.601992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:24.602033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:24.606283 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:24.607122 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:24.607130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:24.607136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:24.607412 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:24.607811 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:24.607815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:24.607817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:24.608363 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:24.608804 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:24.609161 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 10:26:24.609837 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:24.617743 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.609342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:37831)
0925 10:26:24.609679 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.609694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.609784 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.610074 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.610076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.610357 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.610544 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.610554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.611928 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.611933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.613192 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.613208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.614049 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.614055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.614271 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.614858 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.614874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.617525 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227}
W20260925 10:26:24.617894 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.609424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37831 (local address 127.12.248.126:43993)
0925 10:26:24.609666 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.609671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.609761 (+ 90us) server_negotiation.cc:407] Connection header received
0925 10:26:24.609872 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.609876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.609947 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.610051 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.610691 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.611784 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.613344 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.613899 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.614015 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.617690 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:37831: BlockingRecv error: recv got EOF from 127.12.248.67:37831 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
W20260925 10:26:24.618113 22570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:24 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:24.712733 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:24.704543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:53777)
0925 10:26:24.705031 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:24.705047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:24.705159 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:24.705476 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:24.705480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:24.705960 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:24.706154 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.706162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.707100 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.707103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.708044 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:24.708055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.708755 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.708760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:24.708896 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.709566 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:24.709598 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:24.712510 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
W20260925 10:26:24.712968 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:24.704581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:53777 (local address 127.12.248.126:43993)
0925 10:26:24.704798 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:24.704807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:24.705127 (+ 320us) server_negotiation.cc:407] Connection header received
0925 10:26:24.705253 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:24.705257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:24.705327 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:24.705406 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:24.706279 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.706975 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.708194 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:24.708611 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:24.708698 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:24.712718 (+ 4020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:53777: BlockingRecv error: recv got EOF from 127.12.248.65:53777 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:25.175765 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:25.167015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:33497)
0925 10:26:25.167535 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:25.167551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:25.167665 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:25.167974 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:25.167978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:25.168428 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:25.168618 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.168626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.169637 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.169640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.170572 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.170583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.171320 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.171329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.171465 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.172126 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:25.172156 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:25.175437 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
W20260925 10:26:25.175931 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:25.167081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:33497 (local address 127.12.248.126:43993)
0925 10:26:25.167262 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:25.167267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:25.167627 (+ 360us) server_negotiation.cc:407] Connection header received
0925 10:26:25.167765 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:25.167769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:25.167830 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:25.167914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:25.168754 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.169510 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.170685 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.171108 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.171198 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.175661 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:33497: BlockingRecv error: recv got EOF from 127.12.248.66:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:25.634536 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:25.618839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:38671)
0925 10:26:25.619475 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:25.619495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:25.619671 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:25.620025 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:25.620030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:25.620600 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:25.620851 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.620863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.622093 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.622099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.623235 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.623250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.624007 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.624014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.624201 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.631252 (+ 7051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:25.631279 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:25.634271 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":321,"threads_started":1}
W20260925 10:26:25.634773 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:25.618882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38671 (local address 127.12.248.126:43993)
0925 10:26:25.619102 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:25.619110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:25.619613 (+ 503us) server_negotiation.cc:407] Connection header received
0925 10:26:25.619826 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:25.619831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:25.619893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:25.619994 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:25.620993 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.621943 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.623387 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.623830 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.623967 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.634541 (+ 10574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:38671: BlockingRecv error: recv got EOF from 127.12.248.67:38671 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:25 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:25.722572 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:25.713779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:46465)
0925 10:26:25.714342 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:25.714358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:25.714519 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:25.714892 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:25.714897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:25.715322 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:25.715491 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.715501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.716760 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.716767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.717954 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:25.717968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.718628 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.718633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:25.718743 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.719383 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:25.719400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:25.722326 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
W20260925 10:26:25.722818 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:25.713856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:46465 (local address 127.12.248.126:43993)
0925 10:26:25.714119 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:25.714127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:25.714446 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:26:25.714654 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:25.714666 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:25.714728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:25.714853 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:25.715635 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.716599 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.718074 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:25.718472 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:25.718577 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:25.722552 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:46465: BlockingRecv error: recv got EOF from 127.12.248.65:46465 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:26.185806 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:26.176935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:40873)
0925 10:26:26.177509 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:26.177524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:26.177672 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:26.178070 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:26.178075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:26.178534 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:26.178697 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.178706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.179760 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.179766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.180640 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.180651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.181358 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.181364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.181565 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.182149 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:26.182166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:26.185562 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":309,"threads_started":1}
W20260925 10:26:26.186069 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:26.177027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:40873 (local address 127.12.248.126:43993)
0925 10:26:26.177225 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:26.177232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:26.177614 (+ 382us) server_negotiation.cc:407] Connection header received
0925 10:26:26.177844 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:26.177848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:26.177928 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:26.178036 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:26.178855 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.179604 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.180787 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.181198 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.181334 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.185758 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:40873: BlockingRecv error: recv got EOF from 127.12.248.66:40873 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:26.644243 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:26.635498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:45057)
0925 10:26:26.636207 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:26.636251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:26.636412 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:26.636802 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:26.636805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:26.637257 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:26.637525 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.637537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.638660 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.638664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.639585 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.639597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.640290 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.640294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.640406 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.640962 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:26.640979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:26.643883 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":394,"threads_started":1}
W20260925 10:26:26.644372 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:26.635740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:45057 (local address 127.12.248.126:43993)
0925 10:26:26.635986 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:26.635994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:26.636361 (+ 367us) server_negotiation.cc:407] Connection header received
0925 10:26:26.636548 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:26.636554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:26.636634 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:26.636736 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:26.637717 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.638510 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.639711 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.640136 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.640228 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.644100 (+ 3872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:45057: BlockingRecv error: recv got EOF from 127.12.248.67:45057 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:26 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:26.731762 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:26.723499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:39061)
0925 10:26:26.724083 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:26.724100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:26.724213 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:26.724506 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:26.724509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:26.724859 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:26.725029 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.725036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.726131 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.726135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.727055 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:26.727067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.727656 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.727660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:26.727778 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.728433 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:26.728449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:26.731401 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":353,"threads_started":1}
W20260925 10:26:26.731918 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:26.723607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:39061 (local address 127.12.248.126:43993)
0925 10:26:26.723820 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:26.723827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:26.724178 (+ 351us) server_negotiation.cc:407] Connection header received
0925 10:26:26.724312 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:26.724316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:26.724378 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:26.724462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:26.725187 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.726000 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.727169 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:26.727540 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:26.727630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:26.731664 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:39061: BlockingRecv error: recv got EOF from 127.12.248.65:39061 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:27.194885 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:27.186728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:49873)
0925 10:26:27.187200 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:27.187216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:27.187332 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:27.187615 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:27.187618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:27.187957 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:27.188135 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.188144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.189135 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.189139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.190151 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.190163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.190858 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.190862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.191062 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.191709 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:27.191726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:27.194619 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
W20260925 10:26:27.195044 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:27.186791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:49873 (local address 127.12.248.126:43993)
0925 10:26:27.187008 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:27.187017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:27.187294 (+ 277us) server_negotiation.cc:407] Connection header received
0925 10:26:27.187429 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:27.187433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:27.187494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:27.187574 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:27.188261 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.189025 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.190281 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.190731 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.190852 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.194828 (+ 3976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:49873: BlockingRecv error: recv got EOF from 127.12.248.66:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:27.654724 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:27.645259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:44489)
0925 10:26:27.645804 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:27.645849 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:27.645977 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:27.646360 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:27.646363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:27.646716 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:27.646906 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.646926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.648171 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.648176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.649135 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.649148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.650045 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.650052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.650285 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.651089 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:27.651117 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:27.654439 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":311,"threads_started":1}
W20260925 10:26:27.654852 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:27.645522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:44489 (local address 127.12.248.126:43993)
0925 10:26:27.645729 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:27.645735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:27.645979 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:26:27.646084 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:27.646096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:27.646169 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:27.646255 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:27.647070 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.647948 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.649316 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.649903 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.650010 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.654643 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:44489: BlockingRecv error: recv got EOF from 127.12.248.67:44489 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:27 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:27.741660 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:27.732834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:36209)
0925 10:26:27.733334 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:27.733349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:27.733537 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:27.733880 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:27.733885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:27.734274 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:27.734426 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.734434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.735493 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.735499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.736915 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:27.736930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.737801 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.737806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:27.737935 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.738599 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:27.738617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:27.741361 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":279,"threads_started":1}
W20260925 10:26:27.741894 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:27.732900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:36209 (local address 127.12.248.126:43993)
0925 10:26:27.733152 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:27.733160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:27.733479 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:26:27.733676 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:27.733680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:27.733741 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:27.733847 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:27.734569 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.735345 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.737061 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:27.737667 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:27.737765 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:27.741646 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:36209: BlockingRecv error: recv got EOF from 127.12.248.65:36209 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:28.204700 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:28.196044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:53265)
0925 10:26:28.196588 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:28.196613 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:28.196728 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:28.197022 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:28.197026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:28.197456 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:28.197640 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.197652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.199081 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.199087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.200159 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.200171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.200936 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.200940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.201059 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.201741 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:28.201758 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:28.204475 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":323,"threads_started":1}
W20260925 10:26:28.204965 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:28.196088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53265 (local address 127.12.248.126:43993)
0925 10:26:28.196288 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:28.196292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:28.196693 (+ 401us) server_negotiation.cc:407] Connection header received
0925 10:26:28.196828 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:28.196832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:28.196893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:28.196976 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:28.197816 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.198933 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.200370 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.200801 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.200900 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.204688 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:53265: BlockingRecv error: recv got EOF from 127.12.248.66:53265 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:28.664269 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:28.655716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:38461)
0925 10:26:28.656259 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:28.656279 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:28.656408 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:28.656763 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:28.656768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:28.657161 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:28.657316 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.657325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.658381 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.658387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.659395 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.659407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.660161 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.660167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.660340 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.660927 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:28.660944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:28.664049 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":316,"threads_started":1}
W20260925 10:26:28.664583 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:28.655757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38461 (local address 127.12.248.126:43993)
0925 10:26:28.655956 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:28.655964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:28.656358 (+ 394us) server_negotiation.cc:407] Connection header received
0925 10:26:28.656541 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:28.656546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:28.656619 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:28.656723 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:28.657484 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.658240 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.659563 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.660007 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.660136 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.664307 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:38461: BlockingRecv error: recv got EOF from 127.12.248.67:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:28 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:28.750898 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:28.742721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:49797)
0925 10:26:28.743199 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:28.743215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:28.743362 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:28.743702 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:28.743706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:28.744190 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:28.744356 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.744366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.745407 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.745413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.746389 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:28.746401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.747128 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.747133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:28.747310 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.747879 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:28.747896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:28.750663 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
W20260925 10:26:28.751235 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:28.742787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49797 (local address 127.12.248.126:43993)
0925 10:26:28.742992 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:28.743000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:28.743314 (+ 314us) server_negotiation.cc:407] Connection header received
0925 10:26:28.743492 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:28.743497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:28.743572 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:28.743672 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:28.744512 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.745246 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.746539 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:28.746973 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:28.747104 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:28.750866 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:49797: BlockingRecv error: recv got EOF from 127.12.248.65:49797 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:29.214080 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:29.205724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:44989)
0925 10:26:29.206640 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:29.206659 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:29.206810 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:29.207176 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:29.207184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:29.207569 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:29.207728 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.207737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.208718 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.208722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.209644 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.209655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.210294 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.210298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.210409 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.210961 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:29.210977 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:29.213821 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":406,"threads_started":1}
W20260925 10:26:29.214358 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:29.206247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:44989 (local address 127.12.248.126:43993)
0925 10:26:29.206457 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:29.206461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:29.206748 (+ 287us) server_negotiation.cc:407] Connection header received
0925 10:26:29.206929 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:29.206933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:29.207021 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:29.207116 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:29.207859 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.208600 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.209765 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.210176 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.210268 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.214024 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:44989: BlockingRecv error: recv got EOF from 127.12.248.66:44989 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:29.674255 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:29.665363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:58611)
0925 10:26:29.665952 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:29.665973 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:29.666134 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:29.666489 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:29.666493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:29.666927 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:29.667084 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.667092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.668227 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.668235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.669382 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.669393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.670208 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.670214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.670403 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.671064 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:29.671082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:29.673890 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":288,"threads_started":1}
W20260925 10:26:29.674341 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:29.665531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:58611 (local address 127.12.248.126:43993)
0925 10:26:29.665771 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:29.665779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:29.666060 (+ 281us) server_negotiation.cc:407] Connection header received
0925 10:26:29.666261 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:29.666267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:29.666337 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:29.666451 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:29.667317 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.668061 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.669589 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.670057 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.670196 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.674096 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:58611: BlockingRecv error: recv got EOF from 127.12.248.67:58611 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:29 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:29.760251 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:29.751868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:37805)
0925 10:26:29.752465 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:29.752480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:29.752659 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:29.753042 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:29.753048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:29.753543 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:29.753692 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.753700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.754748 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.754755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.755637 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:29.755648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.756458 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.756464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:29.756638 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.757268 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:29.757285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:29.760032 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":330,"threads_started":1}
W20260925 10:26:29.760470 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:29.751974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:37805 (local address 127.12.248.126:43993)
0925 10:26:29.752156 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:29.752161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:29.752592 (+ 431us) server_negotiation.cc:407] Connection header received
0925 10:26:29.752788 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:29.752815 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:29.752890 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:29.753001 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:29.753841 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.754606 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.755785 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:29.756297 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:29.756424 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:29.760227 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:37805: BlockingRecv error: recv got EOF from 127.12.248.65:37805 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:30.223601 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:30.215300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:34501)
0925 10:26:30.215811 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:30.215830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:30.215983 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:30.216334 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:30.216339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:30.216738 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:30.216895 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.216903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.218006 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.218012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.218976 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.218987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.219676 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.219682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.219847 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.220520 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:30.220538 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:30.223388 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":375,"threads_started":1}
W20260925 10:26:30.223929 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:30.215426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:34501 (local address 127.12.248.126:43993)
0925 10:26:30.215660 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:30.215667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:30.215925 (+ 258us) server_negotiation.cc:407] Connection header received
0925 10:26:30.216133 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:30.216137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:30.216200 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:30.216300 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:30.217043 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.217873 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.219134 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.219517 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.219656 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.223625 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:34501: BlockingRecv error: recv got EOF from 127.12.248.66:34501 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:30.691615 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:30.675290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:45031)
0925 10:26:30.676058 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:30.676084 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:30.676227 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:30.676728 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:30.676733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:30.677156 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:30.677323 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.677332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.678333 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.678337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.679231 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.679242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.679921 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.679925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.680042 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.688272 (+ 8230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:30.688326 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:30.691403 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":497,"threads_started":1}
W20260925 10:26:30.691871 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:30.675422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:45031 (local address 127.12.248.126:43993)
0925 10:26:30.675627 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:30.675634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:30.676160 (+ 526us) server_negotiation.cc:407] Connection header received
0925 10:26:30.676472 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:30.676478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:30.676562 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:30.676675 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:30.677463 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.678198 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.679365 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.679793 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.679885 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.691592 (+ 11707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:45031: BlockingRecv error: recv got EOF from 127.12.248.67:45031 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:30 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:30.769855 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:30.761227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:60975)
0925 10:26:30.762082 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:30.762099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:30.762210 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:30.762495 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:30.762498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:30.762905 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:30.763067 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.763075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.764139 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.764147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.765143 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:30.765155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.765850 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.765855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:30.765967 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.766540 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:30.766571 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:30.769603 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":490,"threads_started":1}
W20260925 10:26:30.770049 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:30.761279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:60975 (local address 127.12.248.126:43993)
0925 10:26:30.761549 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:30.761559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:30.762189 (+ 630us) server_negotiation.cc:407] Connection header received
0925 10:26:30.762310 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:30.762314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:30.762375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:30.762454 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:30.763187 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.763998 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.765288 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:30.765714 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:30.765812 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:30.769820 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:60975: BlockingRecv error: recv got EOF from 127.12.248.65:60975 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:31.232841 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:31.224553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:58065)
0925 10:26:31.225113 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:31.225129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:31.225281 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:31.225588 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:31.225592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:31.226039 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:31.226204 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.226212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.227206 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.227210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.228136 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.228147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.228756 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.228761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.228868 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.229534 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:31.229551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:31.232585 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":273,"threads_started":1}
W20260925 10:26:31.233007 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:31.224646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:58065 (local address 127.12.248.126:43993)
0925 10:26:31.224985 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:31.224991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:31.225206 (+ 215us) server_negotiation.cc:407] Connection header received
0925 10:26:31.225381 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:31.225384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:31.225466 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:31.225545 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:31.226361 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.227082 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.228253 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.228631 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.228719 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.232754 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:58065: BlockingRecv error: recv got EOF from 127.12.248.66:58065 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:31.701992 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:31.692583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:46809)
0925 10:26:31.693184 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:31.693203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:31.693368 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:31.693720 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:31.693724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:31.694285 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:31.694467 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.694477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.695684 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.695691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.696720 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.696733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.697638 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.697644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.697828 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.698456 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:31.698484 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:31.701623 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":295,"threads_started":1}
W20260925 10:26:31.702097 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:31.692682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:46809 (local address 127.12.248.126:43993)
0925 10:26:31.692917 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:31.692926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:31.693320 (+ 394us) server_negotiation.cc:407] Connection header received
0925 10:26:31.693515 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:31.693521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:31.693588 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:31.693688 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:31.694654 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.695495 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.696882 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.697394 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.697632 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.701798 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:46809: BlockingRecv error: recv got EOF from 127.12.248.67:46809 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:31 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:31.779582 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:31.770835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:57559)
0925 10:26:31.771455 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:31.771471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:31.771632 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:31.772001 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:31.772005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:31.772448 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:31.772595 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.772603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.773683 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.773688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.774683 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:31.774696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.775576 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.775582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:31.775768 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.776411 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:31.776428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:31.779349 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":284,"threads_started":1}
W20260925 10:26:31.779986 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:31.771121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:57559 (local address 127.12.248.126:43993)
0925 10:26:31.771390 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:31.771396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:31.771555 (+ 159us) server_negotiation.cc:407] Connection header received
0925 10:26:31.771794 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:31.771798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:31.771859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:31.771967 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:31.772745 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.773542 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.774837 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:31.775389 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:31.775528 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:31.779610 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:57559: BlockingRecv error: recv got EOF from 127.12.248.65:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:32.242278 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:32.233878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:54679)
0925 10:26:32.234427 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:32.234443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:32.234557 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:32.234868 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:32.234871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:32.235251 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:32.235430 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.235438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.236615 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.236620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.237610 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.237622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.238311 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.238315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.238423 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.239032 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:32.239050 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:32.241878 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
W20260925 10:26:32.242471 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:32.233965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:54679 (local address 127.12.248.126:43993)
0925 10:26:32.234197 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:32.234204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:32.234524 (+ 320us) server_negotiation.cc:407] Connection header received
0925 10:26:32.234657 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:32.234661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:32.234726 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:32.234821 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:32.235553 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.236471 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.237741 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.238144 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.238239 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.242199 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:54679: BlockingRecv error: recv got EOF from 127.12.248.66:54679 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:32.712113 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:32.703200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:51515)
0925 10:26:32.703699 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:32.703719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:32.703923 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:32.704321 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:32.704325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:32.704843 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:32.705033 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.705046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.706402 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.706405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.707298 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.707308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.708070 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.708076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.708238 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.708819 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:32.708836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:32.711767 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
W20260925 10:26:32.712284 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:32.703157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:51515 (local address 127.12.248.126:43993)
0925 10:26:32.703367 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:32.703371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:32.703851 (+ 480us) server_negotiation.cc:407] Connection header received
0925 10:26:32.704059 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:32.704064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:32.704155 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:32.704266 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:32.705222 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.706235 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.707449 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.707894 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.708038 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.712007 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:51515: BlockingRecv error: recv got EOF from 127.12.248.67:51515 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:32 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:32.788902 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:32.780491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:59681)
0925 10:26:32.781026 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:32.781043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:32.781202 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:32.781639 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:32.781644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:32.782056 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:32.782213 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.782222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.783236 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.783242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.784175 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:32.784186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.784913 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.784920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:32.785096 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.785708 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:32.785725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:32.788552 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
W20260925 10:26:32.788992 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:32.780535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:59681 (local address 127.12.248.126:43993)
0925 10:26:32.780814 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:32.780821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:32.781135 (+ 314us) server_negotiation.cc:407] Connection header received
0925 10:26:32.781396 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:32.781400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:32.781479 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:32.781615 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:32.782364 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.783098 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.784327 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:32.784745 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:32.784879 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:32.788749 (+ 3870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:59681: BlockingRecv error: recv got EOF from 127.12.248.65:59681 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:33.252240 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:33.243577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:53469)
0925 10:26:33.244221 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:33.244240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:33.244396 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:33.244787 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:33.244792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:33.245340 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:33.245631 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.245648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.246717 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.246721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.247653 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.247664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.248308 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.248327 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.248444 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.249057 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:33.249073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:33.252007 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":333,"threads_started":1}
W20260925 10:26:33.252430 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:33.243687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53469 (local address 127.12.248.126:43993)
0925 10:26:33.243899 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:33.243908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:33.244380 (+ 472us) server_negotiation.cc:407] Connection header received
0925 10:26:33.244522 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:33.244526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:33.244590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:33.244722 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:33.245817 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.246579 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.247787 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.248181 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.248274 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.252189 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:53469: BlockingRecv error: recv got EOF from 127.12.248.66:53469 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:33.721719 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:33.713127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:35399)
0925 10:26:33.713660 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:33.713680 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:33.713816 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:33.714104 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:33.714107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:33.714473 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:33.714634 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.714642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.716055 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.716062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.717179 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.717192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.717990 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.717994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.718117 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.718684 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:33.718715 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:33.721428 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
W20260925 10:26:33.721861 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:33.713205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:35399 (local address 127.12.248.126:43993)
0925 10:26:33.713518 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:33.713523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:33.713760 (+ 237us) server_negotiation.cc:407] Connection header received
0925 10:26:33.713918 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:33.713921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:33.713983 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:33.714064 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:33.714778 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.715893 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.717308 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.717867 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.717960 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.721652 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:35399: BlockingRecv error: recv got EOF from 127.12.248.67:35399 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:33 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:33.798353 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:33.789904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:55627)
0925 10:26:33.790439 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:33.790455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:33.790562 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:33.790910 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:33.790914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:33.791331 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:33.791517 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.791535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.792710 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.792714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.793681 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:33.793700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.794431 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.794435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:33.794544 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.795139 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:33.795156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:33.798096 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":298,"threads_started":1}
W20260925 10:26:33.798861 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:33.789978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:55627 (local address 127.12.248.126:43993)
0925 10:26:33.790162 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:33.790166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:33.790556 (+ 390us) server_negotiation.cc:407] Connection header received
0925 10:26:33.790661 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:33.790665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:33.790728 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:33.790888 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:33.791696 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.792583 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.793834 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:33.794291 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:33.794397 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:33.798524 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:55627: BlockingRecv error: recv got EOF from 127.12.248.65:55627 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:34.261158 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:34.253257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:45439)
0925 10:26:34.253817 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:34.253833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:34.253940 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:34.254226 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:34.254229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:34.254602 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:34.254738 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.254745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.255735 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.255739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.256694 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.256706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.257381 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.257385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.257559 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.258204 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:34.258222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:34.260933 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":339,"threads_started":1}
W20260925 10:26:34.261495 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:34.253309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:45439 (local address 127.12.248.126:43993)
0925 10:26:34.253542 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:34.253547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:34.253912 (+ 365us) server_negotiation.cc:407] Connection header received
0925 10:26:34.254038 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:34.254041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:34.254101 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:34.254179 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:34.254866 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.255608 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.256817 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.257226 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.257319 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.261178 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:45439: BlockingRecv error: recv got EOF from 127.12.248.66:45439 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:34.731421 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:34.722848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:42671)
0925 10:26:34.723406 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:34.723428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:34.723587 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:34.723952 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:34.723957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:34.724365 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:34.724561 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.724571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.725623 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.725629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.726614 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.726626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.727353 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.727359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.727526 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.728180 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:34.728197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:34.731077 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
W20260925 10:26:34.731580 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:34.722971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:42671 (local address 127.12.248.126:43993)
0925 10:26:34.723158 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:34.723163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:34.723519 (+ 356us) server_negotiation.cc:407] Connection header received
0925 10:26:34.723710 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:34.723718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:34.723803 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:34.723914 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:34.724713 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.725478 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.726766 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.727178 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.727309 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.731320 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:42671: BlockingRecv error: recv got EOF from 127.12.248.67:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:34 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:34.807583 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:34.799256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:42161)
0925 10:26:34.799789 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:34.799809 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:34.799960 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:34.800254 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:34.800257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:34.800695 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:34.800880 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.800889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.801989 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.801993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.802879 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:34.802890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.803562 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.803566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:34.803685 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.804313 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:34.804331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:34.807224 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
W20260925 10:26:34.807690 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:34.799372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:42161 (local address 127.12.248.126:43993)
0925 10:26:34.799590 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:34.799595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:34.799965 (+ 370us) server_negotiation.cc:407] Connection header received
0925 10:26:34.800058 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:34.800061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:34.800125 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:34.800215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:34.801024 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.801869 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.802999 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:34.803434 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:34.803525 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:34.807459 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:42161: BlockingRecv error: recv got EOF from 127.12.248.65:42161 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:35.270588 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:35.262096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:39381)
0925 10:26:35.262596 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:35.262611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:35.262763 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:35.263109 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:35.263114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:35.263543 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:35.263741 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.263755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.264895 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.264902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.265814 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.265837 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.266547 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.266553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.266740 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.267379 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:35.267396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:35.270244 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260925 10:26:35.270699 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:35.262193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:39381 (local address 127.12.248.126:43993)
0925 10:26:35.262431 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:35.262439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:35.262693 (+ 254us) server_negotiation.cc:407] Connection header received
0925 10:26:35.262919 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:35.262923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:35.262982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:35.263082 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:35.263912 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.264731 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.265980 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.266400 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.266534 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.270454 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:39381: BlockingRecv error: recv got EOF from 127.12.248.66:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:35.752602 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:35.732445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:37369)
0925 10:26:35.732921 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:35.732941 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:35.733056 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:35.733384 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:35.733388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:35.733877 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:35.734056 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.734064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.735091 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.735102 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.736423 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.736437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.737380 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.737386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.737583 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.749382 (+ 11799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:35.749413 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:35.752364 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
W20260925 10:26:35.752803 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:35.732583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37369 (local address 127.12.248.126:43993)
0925 10:26:35.732829 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:35.732835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:35.733022 (+ 187us) server_negotiation.cc:407] Connection header received
0925 10:26:35.733161 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:35.733166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:35.733244 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:35.733349 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:35.734234 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.734966 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.736572 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.737185 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.737332 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.752573 (+ 15241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:37369: BlockingRecv error: recv got EOF from 127.12.248.67:37369 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:35 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:35.817061 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:35.808558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:59945)
0925 10:26:35.809231 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:35.809247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:35.809420 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:35.809830 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:35.809835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:35.810350 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:35.810501 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.810510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.811550 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.811556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.812477 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:35.812488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.813262 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.813268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:35.813511 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.814080 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:35.814096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:35.816841 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":445,"threads_started":1}
W20260925 10:26:35.817257 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:35.808669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:59945 (local address 127.12.248.126:43993)
0925 10:26:35.808889 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:35.808895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:35.809370 (+ 475us) server_negotiation.cc:407] Connection header received
0925 10:26:35.809583 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:35.809589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:35.809669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:35.809787 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:35.810647 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.811405 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.812622 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:35.813109 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:35.813256 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:35.817035 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:59945: BlockingRecv error: recv got EOF from 127.12.248.65:59945 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:36.279863 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:36.271696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:36191)
0925 10:26:36.272425 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:36.272445 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:36.272585 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:36.272914 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:36.272918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:36.273292 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:36.273475 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.273484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.274446 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.274450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.275352 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.275362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.276016 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.276020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.276138 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.276793 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:36.276810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:36.279614 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":473,"threads_started":1}
W20260925 10:26:36.280027 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:36.271761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:36191 (local address 127.12.248.126:43993)
0925 10:26:36.272017 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:36.272028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:36.272531 (+ 503us) server_negotiation.cc:407] Connection header received
0925 10:26:36.272711 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:36.272715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:36.272776 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:36.272861 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:36.273605 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.274322 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.275468 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.275892 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.275984 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.279805 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:36191: BlockingRecv error: recv got EOF from 127.12.248.66:36191 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:36.762583 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:36.753611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:39761)
0925 10:26:36.754139 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:36.754159 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:36.754315 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:36.754677 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:36.754682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:36.755112 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:36.755260 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.755268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.756373 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.756380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.757286 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.757297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.758088 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.758094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.758279 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.759083 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:36.759105 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:36.762252 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":320,"threads_started":1}
W20260925 10:26:36.762744 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:36.753670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:39761 (local address 127.12.248.126:43993)
0925 10:26:36.753899 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:36.753906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:36.754249 (+ 343us) server_negotiation.cc:407] Connection header received
0925 10:26:36.754450 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:36.754458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:36.754525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:36.754631 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:36.755410 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.756216 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.757456 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.757940 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.758075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.762485 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:39761: BlockingRecv error: recv got EOF from 127.12.248.67:39761 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:36 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:36.826964 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:36.818153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:44773)
0925 10:26:36.818752 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:36.818768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:36.818971 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:36.819337 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:36.819346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:36.819942 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:36.820126 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.820135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.821476 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.821480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.822420 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:36.822430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.823249 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.823253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:36.823368 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.823955 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:36.823972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:36.826714 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":333,"threads_started":1}
W20260925 10:26:36.827095 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:36.818240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:44773 (local address 127.12.248.126:43993)
0925 10:26:36.818520 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:36.818529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:36.818916 (+ 387us) server_negotiation.cc:407] Connection header received
0925 10:26:36.819105 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:36.819110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:36.819186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:36.819263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:36.820301 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.821316 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.822552 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:36.823090 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:36.823195 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:36.826883 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:44773: BlockingRecv error: recv got EOF from 127.12.248.65:44773 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:37.288892 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:37.280846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:43853)
0925 10:26:37.281377 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:37.281393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:37.281605 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:37.281923 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:37.281927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:37.282333 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:37.282487 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.282496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.283507 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.283510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.284448 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.284460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.285137 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.285142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.285251 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.285897 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:37.285914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:37.288637 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
W20260925 10:26:37.289064 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:37.281054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:43853 (local address 127.12.248.126:43993)
0925 10:26:37.281289 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:37.281296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:37.281514 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:26:37.281712 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:37.281716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:37.281795 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:37.281874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:37.282616 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.283385 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.284576 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.284988 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.285077 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.288853 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:43853: BlockingRecv error: recv got EOF from 127.12.248.66:43853 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:37.771983 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:37.763679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:57323)
0925 10:26:37.764225 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:37.764244 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:37.764423 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:37.764798 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:37.764806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:37.765206 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:37.765360 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.765368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.766422 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.766428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.767344 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.767355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.768081 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.768087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.768248 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.768820 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:37.768836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:37.771579 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":296,"threads_started":1}
W20260925 10:26:37.772030 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:37.763780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:57323 (local address 127.12.248.126:43993)
0925 10:26:37.763992 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:37.764000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:37.764349 (+ 349us) server_negotiation.cc:407] Connection header received
0925 10:26:37.764546 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:37.764553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:37.764618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:37.764755 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:37.765546 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.766278 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.767495 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.767904 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.768044 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.771820 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:57323: BlockingRecv error: recv got EOF from 127.12.248.67:57323 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:37 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:37.836961 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:37.828001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:55639)
0925 10:26:37.828541 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:37.828558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:37.828673 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:37.828982 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:37.828985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:37.829338 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:37.829600 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.829611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.830732 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.830736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.831657 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:37.831668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.832386 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.832391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:37.832510 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.833149 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:37.833166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:37.836601 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":307,"threads_started":1}
W20260925 10:26:37.837074 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:37.828148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:55639 (local address 127.12.248.126:43993)
0925 10:26:37.828472 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:37.828478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:37.828644 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:26:37.828777 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:37.828780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:37.828854 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:37.828937 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:37.829742 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.830592 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.831797 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:37.832247 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:37.832344 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:37.836809 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:55639: BlockingRecv error: recv got EOF from 127.12.248.65:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":188}
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:38.299034 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:38.289975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:56905)
0925 10:26:38.290514 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:38.290531 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:38.290640 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:38.290985 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:38.290989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:38.291451 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:38.291663 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.291673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.292734 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.292739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.293845 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.293859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.294611 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.294618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.294810 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.295596 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:38.295613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:38.298783 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":237,"threads_started":1}
W20260925 10:26:38.299327 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:38.290028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:56905 (local address 127.12.248.126:43993)
0925 10:26:38.290336 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:38.290343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:38.290589 (+ 246us) server_negotiation.cc:407] Connection header received
0925 10:26:38.290740 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:38.290744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:38.290809 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:38.290896 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:38.291798 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.292552 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.293982 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.294447 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.294593 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.299014 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:56905: BlockingRecv error: recv got EOF from 127.12.248.66:56905 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.67@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:38.781816 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:38.772954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.67:54929)
0925 10:26:38.773579 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:38.773601 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:38.773756 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:38.774118 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:38.774123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:38.774553 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:38.774737 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.774745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.775806 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.775811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.776816 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.776827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.777620 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.777625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.777821 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.778499 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:38.778516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:38.781409 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":330,"threads_started":1}
W20260925 10:26:38.782212 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:38.773115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:54929 (local address 127.12.248.126:43993)
0925 10:26:38.773361 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:38.773370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:38.773685 (+ 315us) server_negotiation.cc:407] Connection header received
0925 10:26:38.773887 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:38.773892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:38.773978 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:38.774085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:38.774883 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.775637 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.776962 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.777395 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.777610 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.781698 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.67:54929: BlockingRecv error: recv got EOF from 127.12.248.67:54929 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:38 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.65@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:38.847039 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:38.838383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.65:39599)
0925 10:26:38.839033 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:38.839057 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:38.839219 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:38.839600 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:38.839603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:38.840010 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:38.840173 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.840181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.841153 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.841157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.842301 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:38.842318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.843130 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.843134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:38.843314 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.843936 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:38.843952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:38.846765 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
W20260925 10:26:38.847167 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:38.838446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:39599 (local address 127.12.248.126:43993)
0925 10:26:38.838670 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:38.838680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:38.839167 (+ 487us) server_negotiation.cc:407] Connection header received
0925 10:26:38.839396 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:38.839402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:38.839471 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:38.839549 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:38.840315 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.841031 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.842503 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:38.842987 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:38.843120 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:38.846981 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.65:39599: BlockingRecv error: recv got EOF from 127.12.248.65:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 10:26:39 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:26:39 dist-test-slave-f9rl krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.248.66@KRBTEST.COM for oryx/127.12.248.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:26:39.308691 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:39.300300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43993 (local address 127.12.248.66:41897)
0925 10:26:39.300903 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:39.300922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:39.301045 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:39.301525 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:39.301537 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:39.301964 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:39.302128 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:39.302136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.303200 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.303205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:39.304120 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:39.304131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.304798 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.304802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:39.304920 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:39.305683 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:39.305700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:39.308427 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.248.126:43993: Server oryx/127.12.248.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
W20260925 10:26:39.308866 22309 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:39.300357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:41897 (local address 127.12.248.126:43993)
0925 10:26:39.300600 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:39.300609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:39.301023 (+ 414us) server_negotiation.cc:407] Connection header received
0925 10:26:39.301240 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:39.301248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:39.301337 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:39.301420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:39.302267 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.303025 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.304245 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.304680 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.304771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:39.308620 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.248.66:41897: BlockingRecv error: recv got EOF from 127.12.248.66:41897 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:26:39.594302 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22160
I20260925 10:26:39.606956 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:39.595508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.12.248.65:34601)
0925 10:26:39.596141 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:39.596148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:39.596176 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:26:39.596239 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:39.596246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:39.596471 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:39.596634 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:39.598044 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.599190 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.600347 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.600903 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.601007 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:39.603541 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:39.603552 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:39.603567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:39.603609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:39.606343 (+ 2734us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:26:39.606622 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45624: Request ticket server kudu/127.12.248.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":331,"threads_started":1}
W20260925 10:26:39.607015 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22160: Not authorized: Client connection negotiation failed: client connection to 127.12.248.65:34601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:26:39.607088 22620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45624: Request ticket server kudu/127.12.248.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:26:39.607242 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22160
I20260925 10:26:39.624310 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22311
I20260925 10:26:39.636337 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:39.625380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.12.248.66:43531)
0925 10:26:39.626094 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:39.626113 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:39.626145 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:26:39.626206 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:39.626212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:39.626441 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:39.626600 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:39.627787 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.628873 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.630096 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.630660 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.630768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:39.633122 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:39.633130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:39.633144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:39.633180 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:39.635828 (+ 2648us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:26:39.636077 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53022: Request ticket server kudu/127.12.248.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":390,"threads_started":1}
W20260925 10:26:39.636510 22621 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53022: Request ticket server kudu/127.12.248.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:26:39.636539 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22311: Not authorized: Client connection negotiation failed: client connection to 127.12.248.66:43531: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:26:39.636698 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22311
I20260925 10:26:39.653571 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22442
I20260925 10:26:39.665927 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:39.654592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.12.248.67:39233)
0925 10:26:39.655289 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:39.655297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:39.655326 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:26:39.655388 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:39.655394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:39.655621 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:39.655783 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:39.657066 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.658205 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.659398 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:39.659999 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:39.660128 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:39.662677 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:39.662686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:39.662703 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:39.662736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:39.665311 (+ 2575us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 10:26:39.665669 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37352: Request ticket server kudu/127.12.248.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":354,"threads_started":1}
W20260925 10:26:39.666055 22622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37352: Request ticket server kudu/127.12.248.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 10:26:39.666131 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22442: Not authorized: Client connection negotiation failed: client connection to 127.12.248.67:39233: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.248.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 10:26:39.666402 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22442
I20260925 10:26:39.682739 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22091
I20260925 10:26:39.684092 22153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:39.801113 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22091
2026-09-25T10:26:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21221 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:39Z Disabled control of system clock
I20260925 10:26:39.881181 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43463
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:26:40.231696 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:40.232016 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:40.232189 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:40.239526 22627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:40.239670 22627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:40.239741 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:40.239846 22627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:40.239933 22627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:40.249331 22627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:43463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:40.250706 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:40.252566 22627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:40.265177 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:40.265309 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:40.265256 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:40.265560 22627 server_base.cc:1034] running on GCE node
I20260925 10:26:40.266687 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:40.268181 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:40.269410 22627 hybrid_clock.cc:648] HybridClock initialized: now 1790332000269362 us; error 36 us; skew 500 ppm
I20260925 10:26:40.269855 22627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:40.272135 22627 webserver.cc:460] Webserver started at http://127.12.248.126:38457/ using document root <none> and password file <none>
I20260925 10:26:40.272904 22627 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:40.273021 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:40.273379 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:40.276511 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "91144720587848cb886afa8097a7bae7"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "744c0fce49e7de3f0ef03e3f2c142fa1"
server_key_iv: "c32780282fdff4a456713520c4a81510"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.277287 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "91144720587848cb886afa8097a7bae7"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "744c0fce49e7de3f0ef03e3f2c142fa1"
server_key_iv: "c32780282fdff4a456713520c4a81510"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.282565 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 10:26:40.286121 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:40.287494 22627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:26:40.287678 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "91144720587848cb886afa8097a7bae7"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "744c0fce49e7de3f0ef03e3f2c142fa1"
server_key_iv: "c32780282fdff4a456713520c4a81510"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.287916 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:40.304867 22627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:40.305893 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:40.306202 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:40.324846 22627 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:43463
I20260925 10:26:40.324865 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:43463 every 8 connection(s)
I20260925 10:26:40.326332 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:40.330209 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:40.334257 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22627
I20260925 10:26:40.334510 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:40.335191 13281 external_mini_cluster.cc:1428] Setting key 5e6625e463cdf41524da1415063e058b
I20260925 10:26:40.342110 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7: Bootstrap starting.
I20260925 10:26:40.345861 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:40.347275 22693 log.cc:824] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:40.350414 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:40.337107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.12.248.126:43463)
0925 10:26:40.338097 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:40.338109 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:40.338154 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:26:40.340474 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:40.340500 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:40.340986 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:40.341330 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:40.342049 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.343411 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.345298 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.346053 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.346255 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:40.347675 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:40.347683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:40.347697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:40.348524 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:40.348641 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:40.348670 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:40.348788 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:40.349034 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:26:40.349386 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":386,"threads_started":1}
I20260925 10:26:40.350749 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7: No bootstrap required, opened a new log
I20260925 10:26:40.359297 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } }
I20260925 10:26:40.359704 22693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:40.359829 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91144720587848cb886afa8097a7bae7, State: Initialized, Role: FOLLOWER
I20260925 10:26:40.360450 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } }
I20260925 10:26:40.360771 22693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:40.360939 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:40.361143 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:40.363454 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } }
I20260925 10:26:40.364010 22693 leader_election.cc:302] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91144720587848cb886afa8097a7bae7; no voters:
I20260925 10:26:40.364710 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:40.364842 22698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:40.365594 22698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [term 1 LEADER]: Becoming Leader. State: Replica: 91144720587848cb886afa8097a7bae7, State: Running, Role: LEADER
I20260925 10:26:40.366124 22698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } }
I20260925 10:26:40.366910 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:40.369081 22700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91144720587848cb886afa8097a7bae7. Latest consensus state: current_term: 1 leader_uuid: "91144720587848cb886afa8097a7bae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } } }
I20260925 10:26:40.369475 22700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:40.370371 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91144720587848cb886afa8097a7bae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91144720587848cb886afa8097a7bae7" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43463 } } }
I20260925 10:26:40.370741 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:40.371840 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:40.376842 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:40.386333 22705 catalog_manager.cc:1347] Generated new cluster ID: b167b24c9f4745d7aab52296ba503189
I20260925 10:26:40.386500 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:40.395009 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:40.396574 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:40.404711 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91144720587848cb886afa8097a7bae7: Generated new TSK 0
I20260925 10:26:40.405697 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:26:40.409951 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--builtin_ntp_servers=127.12.248.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:26:40.742799 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:40.743098 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:40.743211 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:40.750757 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:40.750933 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:40.751124 22717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:40.760345 22717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:40.761788 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:40.763587 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:40.776991 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:40.777068 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:40.777091 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:40.778229 22717 server_base.cc:1034] running on GCE node
I20260925 10:26:40.778898 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:40.779866 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:40.781059 22717 hybrid_clock.cc:648] HybridClock initialized: now 1790332000781004 us; error 50 us; skew 500 ppm
I20260925 10:26:40.781401 22717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:40.788599 22717 webserver.cc:460] Webserver started at http://127.12.248.65:44917/ using document root <none> and password file <none>
I20260925 10:26:40.789340 22717 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:40.789484 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:40.789835 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:40.792972 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "3c660e61078f44d6b62c077eea2aface"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "9c4d00b11326dcc70e82e442254b4c1a"
server_key_iv: "e53e7ec74f61fdd024b86857da96139f"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.793874 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "3c660e61078f44d6b62c077eea2aface"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "9c4d00b11326dcc70e82e442254b4c1a"
server_key_iv: "e53e7ec74f61fdd024b86857da96139f"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.799240 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 10:26:40.802762 22731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:40.804103 22717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:26:40.804289 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "3c660e61078f44d6b62c077eea2aface"
format_stamp: "Formatted at 2026-09-25 10:26:40 on dist-test-slave-f9rl"
server_key: "9c4d00b11326dcc70e82e442254b4c1a"
server_key_iv: "e53e7ec74f61fdd024b86857da96139f"
server_key_version: "encryptionkey@0"
I20260925 10:26:40.804518 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:40.823459 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:40.824374 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:40.824684 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:40.825757 22717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:40.827314 22717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:40.827405 22717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:40.827561 22717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:40.827659 22717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:40.860637 22717 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:33201
I20260925 10:26:40.860714 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:33201 every 8 connection(s)
I20260925 10:26:40.862177 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:40.862969 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22717
I20260925 10:26:40.863197 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:40.864019 13281 external_mini_cluster.cc:1428] Setting key b6672a9b390cf6ed24a8ce680f616630
I20260925 10:26:40.867281 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--builtin_ntp_servers=127.12.248.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:26:40.877022 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:40.865163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:42399 (local address 127.12.248.126:43463)
0925 10:26:40.865686 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:40.865691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:40.867295 (+ 1604us) server_negotiation.cc:407] Connection header received
0925 10:26:40.869159 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:40.869167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:40.869269 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:40.869396 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:40.870555 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.871792 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.873690 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.874262 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.874378 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:40.876148 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:40.876152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:40.876155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:40.876176 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:40.876216 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:40.876228 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:40.876298 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:40.876672 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 10:26:40.876755 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
I20260925 10:26:40.878480 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:40.866013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43463 (local address 127.12.248.65:42399)
0925 10:26:40.867123 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:40.867170 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:40.868876 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:40.869670 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:40.869678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:40.869771 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:40.870375 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:40.870394 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.871983 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.871987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:40.873546 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:40.873561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:40.874523 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:40.874531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:40.874883 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:40.875734 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:40.875740 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:40.875813 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:40.875930 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:40.876328 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:40.876342 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:40.876354 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 10:26:40.876533 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 10:26:40.876996 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":362,"threads_started":1}
I20260925 10:26:40.880085 22844 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43463
I20260925 10:26:40.880409 22844 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:40.881287 22844 heartbeater.cc:511] Master 127.12.248.126:43463 requested a full tablet report, sending...
I20260925 10:26:40.883419 22658 ts_manager.cc:194] Registered new tserver with Master: 3c660e61078f44d6b62c077eea2aface (127.12.248.65:33201)
I20260925 10:26:40.884981 22658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.65:42399
W20260925 10:26:41.209571 22848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:41.209939 22848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:41.210120 22848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:41.217988 22848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:41.218192 22848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:41.218380 22848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:41.228111 22848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:41.229604 22848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:41.231550 22848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:41.245782 22856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:41.245893 22854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:41.245867 22853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:41.246649 22848 server_base.cc:1034] running on GCE node
I20260925 10:26:41.247257 22848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:41.248184 22848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:41.249383 22848 hybrid_clock.cc:648] HybridClock initialized: now 1790332001249318 us; error 60 us; skew 500 ppm
I20260925 10:26:41.249732 22848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:41.252130 22848 webserver.cc:460] Webserver started at http://127.12.248.66:45775/ using document root <none> and password file <none>
I20260925 10:26:41.252853 22848 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:41.252969 22848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:41.253304 22848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:41.256202 22848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "c8e9f9877d134e3aae1bdab219536ac7"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "d458336c3a58bf642a257de86f20a77b"
server_key_iv: "82cd10c3db19db29adb36686044555dc"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.257021 22848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "c8e9f9877d134e3aae1bdab219536ac7"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "d458336c3a58bf642a257de86f20a77b"
server_key_iv: "82cd10c3db19db29adb36686044555dc"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.262491 22848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:26:41.266386 22862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.267905 22848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:26:41.268116 22848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "c8e9f9877d134e3aae1bdab219536ac7"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "d458336c3a58bf642a257de86f20a77b"
server_key_iv: "82cd10c3db19db29adb36686044555dc"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.268364 22848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:41.284276 22848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:41.285152 22848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:41.285531 22848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:41.286522 22848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:41.287998 22848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:41.288129 22848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.288255 22848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:41.288350 22848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.321516 22848 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:33507
I20260925 10:26:41.321579 22974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:33507 every 8 connection(s)
I20260925 10:26:41.323007 22848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:41.333247 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22848
I20260925 10:26:41.333560 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:41.334087 13281 external_mini_cluster.cc:1428] Setting key fe7219461072954e000f57c2450a8d51
I20260925 10:26:41.337318 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--builtin_ntp_servers=127.12.248.84:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:26:41.339049 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.325816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:42117 (local address 127.12.248.126:43463)
0925 10:26:41.326043 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:41.326050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:41.327670 (+ 1620us) server_negotiation.cc:407] Connection header received
0925 10:26:41.330275 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:41.330286 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:41.330661 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:41.330840 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:41.332187 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.333084 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.335473 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.336092 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.336232 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.337961 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:41.337971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:41.337979 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:41.338005 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:41.338055 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:41.338075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:41.338185 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:41.338503 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 10:26:41.338622 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:26:41.340572 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.326445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43463 (local address 127.12.248.66:42117)
0925 10:26:41.327486 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:41.327536 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:41.329845 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:41.331075 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:41.331089 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:41.331211 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:41.331992 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:41.332014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.333271 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.333279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:41.335325 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:41.335340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.336333 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.336343 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:41.336659 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.337562 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:41.337567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:41.337622 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:41.337721 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:41.338232 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:41.338248 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:41.338257 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:26:41.338447 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 10:26:41.338899 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":306,"threads_started":1}
I20260925 10:26:41.342120 22975 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43463
I20260925 10:26:41.342456 22975 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:41.343380 22975 heartbeater.cc:511] Master 127.12.248.126:43463 requested a full tablet report, sending...
I20260925 10:26:41.344954 22658 ts_manager.cc:194] Registered new tserver with Master: c8e9f9877d134e3aae1bdab219536ac7 (127.12.248.66:33507)
I20260925 10:26:41.345925 22658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.66:42117
W20260925 10:26:41.666451 22978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:41.666842 22978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:41.666994 22978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:41.674693 22978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:41.674799 22978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:41.674983 22978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:41.684332 22978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:41.685719 22978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:41.687546 22978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:41.700503 22983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:41.700538 22986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:41.700587 22984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:41.707158 22978 server_base.cc:1034] running on GCE node
I20260925 10:26:41.707952 22978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:41.709141 22978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:41.710373 22978 hybrid_clock.cc:648] HybridClock initialized: now 1790332001710312 us; error 53 us; skew 500 ppm
I20260925 10:26:41.710708 22978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:41.712947 22978 webserver.cc:460] Webserver started at http://127.12.248.67:43623/ using document root <none> and password file <none>
I20260925 10:26:41.713749 22978 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:41.713874 22978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:41.714207 22978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:41.717056 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "2f432bc0b6b941b98a5af65c9bc0cbd3"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "a39257ccd534358eeb912ee7dafbbf19"
server_key_iv: "f1dafb12e3e9bd1936b38db4cf0ec2ad"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.717922 22978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "2f432bc0b6b941b98a5af65c9bc0cbd3"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "a39257ccd534358eeb912ee7dafbbf19"
server_key_iv: "f1dafb12e3e9bd1936b38db4cf0ec2ad"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.722990 22978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:26:41.726336 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.727649 22978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:41.727835 22978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "2f432bc0b6b941b98a5af65c9bc0cbd3"
format_stamp: "Formatted at 2026-09-25 10:26:41 on dist-test-slave-f9rl"
server_key: "a39257ccd534358eeb912ee7dafbbf19"
server_key_iv: "f1dafb12e3e9bd1936b38db4cf0ec2ad"
server_key_version: "encryptionkey@0"
I20260925 10:26:41.728062 22978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:41.741385 22978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:41.742352 22978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:41.742659 22978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:41.743616 22978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:41.745038 22978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:41.745165 22978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.745308 22978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:41.745397 22978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:41.778956 22978 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:43255
I20260925 10:26:41.779047 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:43255 every 8 connection(s)
I20260925 10:26:41.780331 22978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:41.783389 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 22978
I20260925 10:26:41.783571 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:41.784168 13281 external_mini_cluster.cc:1428] Setting key 89b87de6ff1e1fa4c1bb04cdf0d19533
I20260925 10:26:41.793133 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.782905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:41869 (local address 127.12.248.126:43463)
0925 10:26:41.783121 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:41.783126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:41.784762 (+ 1636us) server_negotiation.cc:407] Connection header received
0925 10:26:41.786859 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:41.786865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:41.786952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:41.787074 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:41.788139 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.788906 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.790558 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.790962 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.791112 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.792441 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:41.792446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:41.792449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:41.792467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:41.792499 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:41.792511 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:41.792582 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:41.792785 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:26:41.792882 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:26:41.794148 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.783539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43463 (local address 127.12.248.67:41869)
0925 10:26:41.784585 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:41.784629 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:41.786591 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:41.787303 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:41.787319 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:41.787407 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:41.787974 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:41.787991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.789071 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.789075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:41.790411 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:41.790423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.791211 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.791216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:41.791431 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.792065 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:41.792068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:41.792106 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:41.792244 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:41.792595 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:41.792602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:41.792607 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:26:41.792753 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:26:41.793049 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260925 10:26:41.795568 23105 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43463
I20260925 10:26:41.795873 23105 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:41.796629 23105 heartbeater.cc:511] Master 127.12.248.126:43463 requested a full tablet report, sending...
I20260925 10:26:41.798135 22658 ts_manager.cc:194] Registered new tserver with Master: 2f432bc0b6b941b98a5af65c9bc0cbd3 (127.12.248.67:43255)
I20260925 10:26:41.799098 22658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.67:41869
I20260925 10:26:41.799729 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:26:41.805660 22846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:41.804163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.12.248.126:43463)
0925 10:26:41.804347 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:41.804352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:41.804791 (+ 439us) server_negotiation.cc:407] Connection header received
0925 10:26:41.804928 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:41.804931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:41.804999 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:41.805088 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:41.805429 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58900: BlockingRecv error: recv got EOF from 127.0.0.1:58900 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:26:41.806428 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22717
I20260925 10:26:41.814489 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.807392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.12.248.65:33201)
0925 10:26:41.807988 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:41.808000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:41.808047 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:26:41.808124 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:41.808130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:41.808348 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:41.808467 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:41.808815 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.809924 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.811104 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.811641 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.811739 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.812775 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:41.812783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:41.812797 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:41.813515 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:41.813565 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:41.813585 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:41.813729 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:41.813950 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:26:41.814121 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":322,"threads_started":1}
I20260925 10:26:41.815794 22839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:41.896137 22844 heartbeater.cc:503] Master 127.12.248.126:43463 was elected leader, sending a full tablet report...
I20260925 10:26:41.898053 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22717
I20260925 10:26:41.911487 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22848
I20260925 10:26:41.919386 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:41.912431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.12.248.66:33507)
0925 10:26:41.912912 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:41.912919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:41.912946 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:26:41.913005 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:41.913011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:41.913225 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:41.913356 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:41.913761 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.914710 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.915904 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:41.916444 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:41.916538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:41.917649 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:41.917659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:41.917674 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:41.918429 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:41.918484 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:41.918518 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:41.918632 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:41.918756 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:26:41.918909 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260925 10:26:41.920711 22970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:42.005618 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22848
I20260925 10:26:42.022006 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22978
I20260925 10:26:42.030616 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:42.023095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.12.248.67:43255)
0925 10:26:42.023731 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:42.023738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:42.023767 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:26:42.023836 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:42.023843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:42.024057 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:42.024235 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:42.024708 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:42.025813 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:42.027111 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:42.027688 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:42.027831 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:42.028901 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:42.028910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:42.028924 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:42.029746 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:42.029795 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:42.029823 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:42.029950 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:42.030103 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:26:42.030250 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":342,"threads_started":1}
I20260925 10:26:42.032055 23100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:42.115911 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22978
I20260925 10:26:42.131995 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 22627
I20260925 10:26:42.133234 22688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:42.237082 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 22627
2026-09-25T10:26:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2430 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:42Z Disabled control of system clock
I20260925 10:26:42.308905 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44571
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:26:42.639102 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:42.639428 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:42.639515 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:42.647310 23124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:42.647413 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:42.647450 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:42.647512 23124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:42.647557 23124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:42.656818 23124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:42.658222 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:42.660100 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:42.671912 23129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:42.672070 23130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:42.671923 23132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:42.672928 23124 server_base.cc:1034] running on GCE node
I20260925 10:26:42.673810 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:42.675241 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:42.676448 23124 hybrid_clock.cc:648] HybridClock initialized: now 1790332002676390 us; error 51 us; skew 500 ppm
I20260925 10:26:42.676802 23124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:42.679189 23124 webserver.cc:460] Webserver started at http://127.12.248.126:39713/ using document root <none> and password file <none>
I20260925 10:26:42.679908 23124 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:42.680023 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:42.680385 23124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:42.683286 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "0a42fc024d4145e3ba50accd59216230"
format_stamp: "Formatted at 2026-09-25 10:26:42 on dist-test-slave-f9rl"
server_key: "47f11bc5f5df9fa6217932c5809991e0"
server_key_iv: "814e579a651612fd627856a5ccc358c1"
server_key_version: "encryptionkey@0"
I20260925 10:26:42.684111 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "0a42fc024d4145e3ba50accd59216230"
format_stamp: "Formatted at 2026-09-25 10:26:42 on dist-test-slave-f9rl"
server_key: "47f11bc5f5df9fa6217932c5809991e0"
server_key_iv: "814e579a651612fd627856a5ccc358c1"
server_key_version: "encryptionkey@0"
I20260925 10:26:42.689231 23124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:26:42.692581 23138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:42.693946 23124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:42.694121 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "0a42fc024d4145e3ba50accd59216230"
format_stamp: "Formatted at 2026-09-25 10:26:42 on dist-test-slave-f9rl"
server_key: "47f11bc5f5df9fa6217932c5809991e0"
server_key_iv: "814e579a651612fd627856a5ccc358c1"
server_key_version: "encryptionkey@0"
I20260925 10:26:42.694348 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:42.700261 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:42.701057 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:42.701370 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:42.719825 23124 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:44571
I20260925 10:26:42.719856 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:44571 every 8 connection(s)
I20260925 10:26:42.721191 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:42.724821 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:42.730561 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23124
I20260925 10:26:42.730821 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:42.731462 13281 external_mini_cluster.cc:1428] Setting key 6ddb31efdff5b58c0b5318efaab3bbca
I20260925 10:26:42.736146 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230: Bootstrap starting.
I20260925 10:26:42.740262 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:42.741065 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:42.733590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.12.248.126:44571)
0925 10:26:42.734657 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:42.734674 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:42.734729 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:26:42.737035 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:42.737063 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:42.737605 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:42.738075 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:42.739102 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:42.739111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:42.739126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:42.739143 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:42.739215 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:42.739239 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:42.739440 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:42.739567 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 10:26:42.739807 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":311,"threads_started":1}
I20260925 10:26:42.741690 23190 log.cc:824] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:42.744692 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230: No bootstrap required, opened a new log
I20260925 10:26:42.751856 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } }
I20260925 10:26:42.752177 23190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:42.752247 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a42fc024d4145e3ba50accd59216230, State: Initialized, Role: FOLLOWER
I20260925 10:26:42.752704 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } }
I20260925 10:26:42.752902 23190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:42.752995 23190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:42.753110 23190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:42.755231 23190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } }
I20260925 10:26:42.755659 23190 leader_election.cc:302] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a42fc024d4145e3ba50accd59216230; no voters:
I20260925 10:26:42.756409 23190 leader_election.cc:288] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:42.756515 23195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:42.757181 23195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [term 1 LEADER]: Becoming Leader. State: Replica: 0a42fc024d4145e3ba50accd59216230, State: Running, Role: LEADER
I20260925 10:26:42.757778 23195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } }
I20260925 10:26:42.758634 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:42.760690 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a42fc024d4145e3ba50accd59216230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } } }
I20260925 10:26:42.761641 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:42.763913 23204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:42.763350 23197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a42fc024d4145e3ba50accd59216230. Latest consensus state: current_term: 1 leader_uuid: "0a42fc024d4145e3ba50accd59216230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a42fc024d4145e3ba50accd59216230" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44571 } } }
I20260925 10:26:42.764328 23197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230 [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:42.768687 23204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:42.777318 23204 catalog_manager.cc:1347] Generated new cluster ID: 142f2fddaf134f5dab4314ef1b3f16fb
I20260925 10:26:42.777490 23204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:42.795626 23204 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:42.796159 23204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:42.805720 23204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a42fc024d4145e3ba50accd59216230: Generated new TSK 0
I20260925 10:26:42.806775 23204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:26:42.818686 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--builtin_ntp_servers=127.12.248.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:26:43.157362 23214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:43.157799 23214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:43.157954 23214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:43.165606 23214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:43.165745 23214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:43.165949 23214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:43.176553 23214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:43.178004 23214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:43.179817 23214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:43.193740 23219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:43.193691 23222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:43.193957 23220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:43.194353 23214 server_base.cc:1034] running on GCE node
I20260925 10:26:43.195060 23214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:43.196144 23214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:43.197412 23214 hybrid_clock.cc:648] HybridClock initialized: now 1790332003197353 us; error 43 us; skew 500 ppm
I20260925 10:26:43.197886 23214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:43.200572 23214 webserver.cc:460] Webserver started at http://127.12.248.65:44861/ using document root <none> and password file <none>
I20260925 10:26:43.201395 23214 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:43.201608 23214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:43.201979 23214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:43.206988 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "5b6c58b2f3cc470cae8652d15262d940"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "fefeb65d5e87f0890ae4cef290509b2c"
server_key_iv: "86a921eab682df15a17ad38e8ccf7381"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.207815 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "5b6c58b2f3cc470cae8652d15262d940"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "fefeb65d5e87f0890ae4cef290509b2c"
server_key_iv: "86a921eab682df15a17ad38e8ccf7381"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.213231 23214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:26:43.216960 23228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.218477 23214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:43.218637 23214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "5b6c58b2f3cc470cae8652d15262d940"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "fefeb65d5e87f0890ae4cef290509b2c"
server_key_iv: "86a921eab682df15a17ad38e8ccf7381"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.218865 23214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:43.224825 23214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:43.225744 23214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:43.226032 23214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:43.227008 23214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:43.228438 23214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:43.228523 23214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.228680 23214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:43.228757 23214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.264245 23214 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:34005
I20260925 10:26:43.264343 23340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:34005 every 8 connection(s)
I20260925 10:26:43.265726 23214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:43.271234 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23214
I20260925 10:26:43.271492 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:43.272109 13281 external_mini_cluster.cc:1428] Setting key d4d49c7774addaa320cee4d8ba7ab106
I20260925 10:26:43.274739 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:43.268510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:39839 (local address 127.12.248.126:44571)
0925 10:26:43.268995 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:43.269000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:43.270371 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 10:26:43.272277 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:43.272283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:43.272351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:43.272484 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:43.273950 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:43.273955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:43.273960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:43.273980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:43.274033 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:43.274045 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:43.274181 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:43.274384 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:26:43.274453 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260925 10:26:43.275025 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--builtin_ntp_servers=127.12.248.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:26:43.275794 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:43.269183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44571 (local address 127.12.248.65:39839)
0925 10:26:43.270194 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:43.270244 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:43.271994 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:43.272737 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:43.272745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:43.272847 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:43.273583 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:43.273586 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:43.273626 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:43.273772 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:43.274146 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:43.274157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:43.274165 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:26:43.274456 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:26:43.274739 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":325,"threads_started":1}
I20260925 10:26:43.277468 23341 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44571
I20260925 10:26:43.277777 23341 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:43.278412 23341 heartbeater.cc:511] Master 127.12.248.126:44571 requested a full tablet report, sending...
I20260925 10:26:43.280869 23155 ts_manager.cc:194] Registered new tserver with Master: 5b6c58b2f3cc470cae8652d15262d940 (127.12.248.65:34005)
W20260925 10:26:43.611105 23345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:43.611387 23345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:43.611475 23345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:43.618850 23345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:43.618945 23345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:43.619055 23345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:43.628031 23345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:43.629274 23345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:43.631067 23345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:43.644039 23350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:43.644081 23353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:43.644204 23351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:43.645642 23345 server_base.cc:1034] running on GCE node
I20260925 10:26:43.646247 23345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:43.647190 23345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:43.648437 23345 hybrid_clock.cc:648] HybridClock initialized: now 1790332003648392 us; error 62 us; skew 500 ppm
I20260925 10:26:43.648756 23345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:43.650990 23345 webserver.cc:460] Webserver started at http://127.12.248.66:38189/ using document root <none> and password file <none>
I20260925 10:26:43.651722 23345 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:43.651870 23345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:43.652194 23345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:43.655107 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "9808fb186f4a4471a4a38fd14690929a"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "11ef9dc51ba8acec0c5574b2e1ded233"
server_key_iv: "c6df5f3aad307d4eff2f1456f8e64377"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.655920 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "9808fb186f4a4471a4a38fd14690929a"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "11ef9dc51ba8acec0c5574b2e1ded233"
server_key_iv: "c6df5f3aad307d4eff2f1456f8e64377"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.661222 23345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:26:43.664708 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.666163 23345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:43.666347 23345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "9808fb186f4a4471a4a38fd14690929a"
format_stamp: "Formatted at 2026-09-25 10:26:43 on dist-test-slave-f9rl"
server_key: "11ef9dc51ba8acec0c5574b2e1ded233"
server_key_iv: "c6df5f3aad307d4eff2f1456f8e64377"
server_key_version: "encryptionkey@0"
I20260925 10:26:43.666572 23345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:43.672180 23345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:43.672976 23345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:43.673249 23345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:43.674202 23345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:43.675562 23345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:43.675685 23345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.675823 23345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:43.675915 23345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:43.708986 23345 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:33239
I20260925 10:26:43.709036 23471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:33239 every 8 connection(s)
I20260925 10:26:43.710391 23345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:43.718514 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:43.712983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:53789 (local address 127.12.248.126:44571)
0925 10:26:43.713156 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:43.713161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:43.714562 (+ 1401us) server_negotiation.cc:407] Connection header received
0925 10:26:43.716237 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:43.716242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:43.716322 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:43.716433 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:43.717725 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:43.717730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:43.717735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:43.717761 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:43.717793 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:43.717804 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:43.717944 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:43.718134 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:26:43.718204 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:26:43.719558 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:43.713523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44571 (local address 127.12.248.66:53789)
0925 10:26:43.714411 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:43.714446 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:43.715972 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:43.716657 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:43.716664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:43.716754 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:43.717382 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:43.717384 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:43.717420 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:43.717563 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:43.717902 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:43.717912 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:43.717922 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:26:43.718201 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 10:26:43.718449 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
I20260925 10:26:43.719888 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23345
I20260925 10:26:43.720080 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:43.720674 13281 external_mini_cluster.cc:1428] Setting key 3bc5b7ef318286c6267f5e98cbf4f819
I20260925 10:26:43.721127 23472 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44571
I20260925 10:26:43.721407 23472 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:43.722043 23472 heartbeater.cc:511] Master 127.12.248.126:44571 requested a full tablet report, sending...
I20260925 10:26:43.723687 23155 ts_manager.cc:194] Registered new tserver with Master: 9808fb186f4a4471a4a38fd14690929a (127.12.248.66:33239)
I20260925 10:26:43.723839 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--builtin_ntp_servers=127.12.248.84:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:26:44.057998 23475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:44.058346 23475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:44.058519 23475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:44.065872 23475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:44.066016 23475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:44.066202 23475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:44.075827 23475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:44.077198 23475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:44.079111 23475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:44.092622 23481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:44.092695 23483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:44.093153 23480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:44.093675 23475 server_base.cc:1034] running on GCE node
I20260925 10:26:44.094254 23475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:44.095117 23475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:44.096370 23475 hybrid_clock.cc:648] HybridClock initialized: now 1790332004096330 us; error 31 us; skew 500 ppm
I20260925 10:26:44.096725 23475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:26:44.099123 23475 webserver.cc:460] Webserver started at http://127.12.248.67:44683/ using document root <none> and password file <none>
I20260925 10:26:44.099874 23475 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:44.099989 23475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:44.100337 23475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:44.103272 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "d32a283e9e2a4fd7b369ae9b9e85e234"
format_stamp: "Formatted at 2026-09-25 10:26:44 on dist-test-slave-f9rl"
server_key: "349797958cbb04190ea2b378fb7cab39"
server_key_iv: "9f53ba20d2c6888c4d8b58e4bdc3c29d"
server_key_version: "encryptionkey@0"
I20260925 10:26:44.104074 23475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "d32a283e9e2a4fd7b369ae9b9e85e234"
format_stamp: "Formatted at 2026-09-25 10:26:44 on dist-test-slave-f9rl"
server_key: "349797958cbb04190ea2b378fb7cab39"
server_key_iv: "9f53ba20d2c6888c4d8b58e4bdc3c29d"
server_key_version: "encryptionkey@0"
I20260925 10:26:44.109369 23475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:26:44.112821 23489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:44.114276 23475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:44.114472 23475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "d32a283e9e2a4fd7b369ae9b9e85e234"
format_stamp: "Formatted at 2026-09-25 10:26:44 on dist-test-slave-f9rl"
server_key: "349797958cbb04190ea2b378fb7cab39"
server_key_iv: "9f53ba20d2c6888c4d8b58e4bdc3c29d"
server_key_version: "encryptionkey@0"
I20260925 10:26:44.114713 23475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:44.120440 23475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:44.121249 23475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:44.121588 23475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:44.122604 23475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:44.124161 23475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:44.124281 23475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:44.124429 23475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:44.124519 23475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:44.157997 23475 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:44651
I20260925 10:26:44.158049 23601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:44651 every 8 connection(s)
I20260925 10:26:44.159372 23475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:44.168015 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:44.162053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:58647 (local address 127.12.248.126:44571)
0925 10:26:44.162229 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:44.162234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:44.163876 (+ 1642us) server_negotiation.cc:407] Connection header received
0925 10:26:44.165763 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:44.165768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:44.165855 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:44.165969 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:44.167254 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:44.167259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:44.167264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:44.167286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:44.167323 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:44.167335 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:44.167477 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:44.167664 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 10:26:44.167734 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:26:44.168639 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23475
I20260925 10:26:44.168831 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:44.169028 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:44.162609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44571 (local address 127.12.248.67:58647)
0925 10:26:44.163705 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:44.163742 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:44.165489 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:44.166205 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:44.166213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:44.166311 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:44.166930 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:26:44.166932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:44.166976 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:26:44.167060 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:44.167435 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:44.167445 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:44.167454 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:26:44.167748 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 10:26:44.168001 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":335,"threads_started":1}
I20260925 10:26:44.169314 13281 external_mini_cluster.cc:1428] Setting key 1ebdbdbfa6912e3324889952d1568113
I20260925 10:26:44.170507 23602 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44571
I20260925 10:26:44.170836 23602 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:44.171429 23602 heartbeater.cc:511] Master 127.12.248.126:44571 requested a full tablet report, sending...
I20260925 10:26:44.172747 23155 ts_manager.cc:194] Registered new tserver with Master: d32a283e9e2a4fd7b369ae9b9e85e234 (127.12.248.67:44651)
I20260925 10:26:44.184664 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:26:44.191057 23343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:26:44.189415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34394 (local address 127.12.248.126:44571)
0925 10:26:44.189669 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:44.189676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:44.190183 (+ 507us) server_negotiation.cc:407] Connection header received
0925 10:26:44.190312 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:44.190316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:44.190370 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:44.190451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:44.190782 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34394: BlockingRecv error: recv got EOF from 127.0.0.1:34394 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 10:26:44.191547 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23214
I20260925 10:26:44.194999 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:44.192574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.12.248.65:34005)
0925 10:26:44.193088 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:44.193094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:44.193120 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:26:44.193179 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:44.193185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:44.193405 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:44.193599 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:44.194427 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:44.194433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:44.194445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:44.194463 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:44.194503 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:44.194517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:44.194618 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:44.194696 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:26:44.194773 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260925 10:26:44.196267 23336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:44.279383 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23214
I20260925 10:26:44.296396 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23345
I20260925 10:26:44.300149 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:44.297429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.12.248.66:33239)
0925 10:26:44.297956 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:44.297964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:44.297990 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:26:44.298049 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:44.298055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:44.298302 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:44.298438 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:44.299457 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:44.299463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:44.299474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:44.299491 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:44.299529 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:44.299542 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:44.299645 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:44.299730 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:26:44.299856 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260925 10:26:44.301352 23467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:44.382661 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23345
I20260925 10:26:44.398741 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23475
I20260925 10:26:44.403597 23615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:44.399811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.12.248.67:44651)
0925 10:26:44.401254 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:44.401261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:44.401301 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:26:44.401364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:44.401370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:44.401673 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:44.401806 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:44.402890 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:44.402896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:44.402909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:26:44.402920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:44.402958 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:26:44.402971 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:44.403100 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:44.403150 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:26:44.403256 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":1123,"threads_started":1}
I20260925 10:26:44.404858 23597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:44.486243 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23475
I20260925 10:26:44.503063 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23124
I20260925 10:26:44.504387 23185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:44.608325 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23124
2026-09-25T10:26:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2371 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:26:44 dist-test-slave-f9rl krb5kdc[23620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:26:44 dist-test-slave-f9rl krb5kdc[23620](info): set up 2 sockets
Sep 25 10:26:44 dist-test-slave-f9rl krb5kdc[23620](info): commencing operation
krb5kdc: starting...
W20260925 10:26:45.711786 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:26:45 dist-test-slave-f9rl krb5kdc[23620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:26:45.950371 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35335
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:46.287925 23633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:46.288235 23633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:46.288396 23633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:46.295836 23633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:26:46.295974 23633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:46.296064 23633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:46.296172 23633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:26:46.296257 23633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:26:46.306041 23633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:46.307384 23633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:46.308705 23633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:46.320808 23638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:46.320765 23641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:46.320730 23639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:46.321736 23633 server_base.cc:1034] running on GCE node
I20260925 10:26:46.322533 23633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:46.323909 23633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:46.325119 23633 hybrid_clock.cc:648] HybridClock initialized: now 1790332006325061 us; error 43 us; skew 500 ppm
I20260925 10:26:46.325536 23633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:46 dist-test-slave-f9rl krb5kdc[23620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332006, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:46.334949 23633 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:26:46.336679 23633 webserver.cc:460] Webserver started at http://127.12.248.126:38347/ using document root <none> and password file <none>
I20260925 10:26:46.337426 23633 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:46.337644 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:46.337966 23633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:46.341063 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "e6c27fcd392042698b1bac3fff8370df"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "9898c12b7eb69ed34808c54f68080500"
server_key_iv: "f2a960261d05dc84bf579f60c8e106ed"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.341893 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "e6c27fcd392042698b1bac3fff8370df"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "9898c12b7eb69ed34808c54f68080500"
server_key_iv: "f2a960261d05dc84bf579f60c8e106ed"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.347438 23633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:26:46.351023 23648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:46.352299 23633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:26:46.352519 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "e6c27fcd392042698b1bac3fff8370df"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "9898c12b7eb69ed34808c54f68080500"
server_key_iv: "f2a960261d05dc84bf579f60c8e106ed"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.352746 23633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:46.373335 23633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:46.377650 23633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:46.377956 23633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:46.396718 23633 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:35335
I20260925 10:26:46.396752 23699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:35335 every 8 connection(s)
I20260925 10:26:46.398168 23633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:26:46.401835 23700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:46.403870 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23633
I20260925 10:26:46.404109 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:26:46.404914 13281 external_mini_cluster.cc:1428] Setting key b2b2eb01549cb4f96222ef6542222f2a
I20260925 10:26:46.414044 23700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: Bootstrap starting.
I20260925 10:26:46.417249 23700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:46.418294 23700 log.cc:824] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:26:46 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:46.421706 23700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: No bootstrap required, opened a new log
I20260925 10:26:46.428056 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:46.407175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.12.248.126:35335)
0925 10:26:46.407977 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:46.407990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:46.408032 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:26:46.409910 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:46.409935 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:46.410355 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:46.410730 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:46.412330 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:46.413692 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:46.415013 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:46.415728 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:46.415945 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:46.420469 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:46.420476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:46.420491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:46.420524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:46.423659 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:46.424535 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:46.424542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:46.424551 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:46.424726 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:46.425136 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:46.425139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:46.425141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:46.425769 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:46.426079 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:46.426537 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 10:26:46.426994 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":347,"threads_started":1}
I20260925 10:26:46.430876 23700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } }
I20260925 10:26:46.431248 23700 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:46.431365 23700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c27fcd392042698b1bac3fff8370df, State: Initialized, Role: FOLLOWER
I20260925 10:26:46.431895 23700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } }
I20260925 10:26:46.432137 23700 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:26:46.432281 23700 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:26:46.432461 23700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:46.434733 23700 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } }
I20260925 10:26:46.435214 23700 leader_election.cc:302] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c27fcd392042698b1bac3fff8370df; no voters:
I20260925 10:26:46.435781 23700 leader_election.cc:288] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:26:46.435920 23705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:46.436614 23705 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [term 1 LEADER]: Becoming Leader. State: Replica: e6c27fcd392042698b1bac3fff8370df, State: Running, Role: LEADER
I20260925 10:26:46.437122 23705 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } }
I20260925 10:26:46.438062 23700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:26:46.440091 23707 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c27fcd392042698b1bac3fff8370df. Latest consensus state: current_term: 1 leader_uuid: "e6c27fcd392042698b1bac3fff8370df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } } }
I20260925 10:26:46.440526 23707 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:46.440567 23706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c27fcd392042698b1bac3fff8370df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c27fcd392042698b1bac3fff8370df" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 35335 } } }
I20260925 10:26:46.440873 23706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df [sys.catalog]: This master's current role is: LEADER
I20260925 10:26:46.442373 23713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:26:46.447989 23713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:26:46.456761 23713 catalog_manager.cc:1347] Generated new cluster ID: cce10704aff84ff498e69b205d17fe2f
I20260925 10:26:46.456954 23713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:26:46.474922 23713 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:26:46.476274 23713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:26:46.484279 23713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: Generated new TSK 0
I20260925 10:26:46.485139 23713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:26:46.543416 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:46.898551 23726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:46.898844 23726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:46.899013 23726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:46.906283 23726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:46.906418 23726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:46.906608 23726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:26:46.916419 23726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:46.917815 23726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:46.919230 23726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:46.932948 23731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:46.932912 23734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:46.933014 23732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:46.934041 23726 server_base.cc:1034] running on GCE node
I20260925 10:26:46.934689 23726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:46.935636 23726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:46.936874 23726 hybrid_clock.cc:648] HybridClock initialized: now 1790332006936832 us; error 54 us; skew 500 ppm
I20260925 10:26:46.937198 23726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:46 dist-test-slave-f9rl krb5kdc[23620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332006, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:46.940632 23726 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:26:46.942262 23726 webserver.cc:460] Webserver started at http://127.12.248.65:37597/ using document root <none> and password file <none>
I20260925 10:26:46.942994 23726 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:46.943099 23726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:46.943405 23726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:46.946444 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "153337bc83c24b6fb9d862a817cb34de"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "94f1b2a350799cb66155f95dc7319f32"
server_key_iv: "39e14f04ea84480f24daeabb31d6bd64"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.947209 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "153337bc83c24b6fb9d862a817cb34de"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "94f1b2a350799cb66155f95dc7319f32"
server_key_iv: "39e14f04ea84480f24daeabb31d6bd64"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.952665 23726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:26:46.956156 23741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:46.957733 23726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:46.957896 23726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "153337bc83c24b6fb9d862a817cb34de"
format_stamp: "Formatted at 2026-09-25 10:26:46 on dist-test-slave-f9rl"
server_key: "94f1b2a350799cb66155f95dc7319f32"
server_key_iv: "39e14f04ea84480f24daeabb31d6bd64"
server_key_version: "encryptionkey@0"
I20260925 10:26:46.958062 23726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:46.978569 23726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:46.982913 23726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:46.983197 23726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:46.984241 23726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:46.985714 23726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:46.985787 23726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:46.985941 23726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:46.986006 23726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:47.019503 23726 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:34639
I20260925 10:26:47.019546 23853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:34639 every 8 connection(s)
I20260925 10:26:47.020836 23726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:26:47.027094 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23726
I20260925 10:26:47.027344 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:26:47.028030 13281 external_mini_cluster.cc:1428] Setting key bedb98897a53b69c4b7fd377ed1bb518
Sep 25 10:26:47 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332006, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:47.043102 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:47.023408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:53409 (local address 127.12.248.126:35335)
0925 10:26:47.023864 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:47.023868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:47.024750 (+ 882us) server_negotiation.cc:407] Connection header received
0925 10:26:47.026628 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:47.026633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:47.026723 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:47.026852 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:47.028716 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.029664 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.031480 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.032235 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.032409 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:47.036224 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:47.036232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:47.036240 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:47.036288 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:47.039555 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:47.040541 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:47.040545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:47.040547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:47.040669 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:47.041177 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:47.041180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:47.041182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:47.041501 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:47.041632 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:47.042571 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 10:26:47.042743 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":250,"threads_started":1}
I20260925 10:26:47.044593 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:47.023815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35335 (local address 127.12.248.65:53409)
0925 10:26:47.024594 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:47.024631 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:47.026354 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:47.027085 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:47.027094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:47.027846 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:47.028503 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:47.028521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.029905 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.029911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:47.031298 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:47.031311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.032473 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.032477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:47.032688 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:47.033406 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:47.033478 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:47.035953 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:47.039707 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:47.039720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:47.039742 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:47.040419 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:47.040784 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:47.040788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:47.040793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:47.041066 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:47.041632 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:47.041643 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:47.042147 (+ 504us) client_negotiation.cc:769] Sending connection context
0925 10:26:47.042494 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 10:26:47.043099 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260925 10:26:47.046314 23854 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35335
I20260925 10:26:47.046680 23854 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:47.047515 23854 heartbeater.cc:511] Master 127.12.248.126:35335 requested a full tablet report, sending...
I20260925 10:26:47.049949 23665 ts_manager.cc:194] Registered new tserver with Master: 153337bc83c24b6fb9d862a817cb34de (127.12.248.65:34639)
I20260925 10:26:47.051421 23665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:53409
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:26:47.084846 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:47.422176 23860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:47.422510 23860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:47.422686 23860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:47.430011 23860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:47.430176 23860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:47.430375 23860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:26:47.440112 23860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:47.441417 23860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:47.442909 23860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:47.456338 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:47.456447 23866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:47.456459 23865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:47.456871 23860 server_base.cc:1034] running on GCE node
I20260925 10:26:47.457839 23860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:47.458947 23860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:47.460222 23860 hybrid_clock.cc:648] HybridClock initialized: now 1790332007460121 us; error 27 us; skew 500 ppm
I20260925 10:26:47.460678 23860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:47 dist-test-slave-f9rl krb5kdc[23620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332007, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:47.464094 23860 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:26:47.465656 23860 webserver.cc:460] Webserver started at http://127.12.248.66:39305/ using document root <none> and password file <none>
I20260925 10:26:47.466382 23860 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:47.466538 23860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:47.466861 23860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:47.470022 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "2626f5b3ba6c494f9b9579641ef8b1cc"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "5d13add8921264ef0b24b5e199723ae9"
server_key_iv: "87f74d0a9d34ae0288e196e8c3254057"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.470810 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "2626f5b3ba6c494f9b9579641ef8b1cc"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "5d13add8921264ef0b24b5e199723ae9"
server_key_iv: "87f74d0a9d34ae0288e196e8c3254057"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.475989 23860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:26:47.479514 23875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:47.480948 23860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:26:47.481139 23860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "2626f5b3ba6c494f9b9579641ef8b1cc"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "5d13add8921264ef0b24b5e199723ae9"
server_key_iv: "87f74d0a9d34ae0288e196e8c3254057"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.481365 23860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:47.504487 23860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:47.509251 23860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:47.509613 23860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:47.510706 23860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:47.512140 23860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:47.512256 23860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:47.512415 23860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:47.512492 23860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:47.546340 23860 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:38263
I20260925 10:26:47.546380 23987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:38263 every 8 connection(s)
I20260925 10:26:47.547665 23860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:26:47.551455 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23860
I20260925 10:26:47.551661 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:26:47.552175 13281 external_mini_cluster.cc:1428] Setting key 773987f2b8384ec5210e9fcbb35810c3
Sep 25 10:26:47 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332007, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:47.569468 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:47.550248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:49767 (local address 127.12.248.126:35335)
0925 10:26:47.550670 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:47.550674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:47.551765 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 10:26:47.554067 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:47.554071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:47.554130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:47.554249 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:47.555849 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.556642 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.558292 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.558715 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.558859 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:47.562461 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:47.562470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:47.562478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:47.562523 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:47.566134 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:47.566880 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:47.566888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:47.566895 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:47.567049 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:47.567462 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:47.567470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:47.567476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:47.567862 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:47.568008 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:47.568678 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 10:26:47.568933 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":216,"threads_started":1}
I20260925 10:26:47.570118 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:47.550826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35335 (local address 127.12.248.66:49767)
0925 10:26:47.551570 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:47.551629 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:47.553759 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:47.554499 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:47.554507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:47.555009 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:47.555638 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:47.555678 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.556791 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.556799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:47.558141 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:47.558154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:47.558950 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:47.558957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:47.559214 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:47.559923 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:47.559946 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:47.562225 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:47.566278 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:47.566285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:47.566302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:47.566724 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:47.567159 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:47.567163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:47.567165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:47.567343 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:47.567978 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:47.567984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:47.568302 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 10:26:47.568524 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 10:26:47.568922 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":316,"threads_started":1}
I20260925 10:26:47.571579 23988 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35335
I20260925 10:26:47.571925 23988 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:47.572870 23988 heartbeater.cc:511] Master 127.12.248.126:35335 requested a full tablet report, sending...
I20260925 10:26:47.574625 23665 ts_manager.cc:194] Registered new tserver with Master: 2626f5b3ba6c494f9b9579641ef8b1cc (127.12.248.66:38263)
I20260925 10:26:47.575467 23665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:49767
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:26:47.604977 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:26:47.931747 23994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:26:47.932036 23994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:26:47.932191 23994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:26:47.939801 23994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:26:47.939944 23994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:26:47.940132 23994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:26:47.950405 23994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:35335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:26:47.951764 23994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:26:47.953114 23994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:26:47.967132 23999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:47.967183 24000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:26:47.967067 24002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:26:47.970968 23994 server_base.cc:1034] running on GCE node
I20260925 10:26:47.971580 23994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:26:47.972530 23994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:26:47.973724 23994 hybrid_clock.cc:648] HybridClock initialized: now 1790332007973656 us; error 58 us; skew 500 ppm
I20260925 10:26:47.974015 23994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:26:47 dist-test-slave-f9rl krb5kdc[23620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332007, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:26:47.977167 23994 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:26:47.978739 23994 webserver.cc:460] Webserver started at http://127.12.248.67:36749/ using document root <none> and password file <none>
I20260925 10:26:47.979489 23994 fs_manager.cc:359] Metadata directory not provided
I20260925 10:26:47.979647 23994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:26:47.979952 23994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:26:47.982933 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "8c63175223f74fcc951d463af794f581"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "b2b94a9ce1a52557c919db7e13abba25"
server_key_iv: "59edd17a13e45ffb602e5df5fff4c948"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.983707 23994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "8c63175223f74fcc951d463af794f581"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "b2b94a9ce1a52557c919db7e13abba25"
server_key_iv: "59edd17a13e45ffb602e5df5fff4c948"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.988957 23994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:26:47.992555 24009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:47.993955 23994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:26:47.994128 23994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "8c63175223f74fcc951d463af794f581"
format_stamp: "Formatted at 2026-09-25 10:26:47 on dist-test-slave-f9rl"
server_key: "b2b94a9ce1a52557c919db7e13abba25"
server_key_iv: "59edd17a13e45ffb602e5df5fff4c948"
server_key_version: "encryptionkey@0"
I20260925 10:26:47.994349 23994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:26:48.013363 23994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:26:48.017606 23994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:26:48.017932 23994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:26:48.019109 23994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:26:48.020586 23994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:26:48.020696 23994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:48.020848 23994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:26:48.020921 23994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:26:48.055014 23854 heartbeater.cc:503] Master 127.12.248.126:35335 was elected leader, sending a full tablet report...
I20260925 10:26:48.055270 23994 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:44429
I20260925 10:26:48.055401 24121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:44429 every 8 connection(s)
I20260925 10:26:48.056814 23994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:26:48.067145 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 23994
I20260925 10:26:48.067356 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:26:48.067890 13281 external_mini_cluster.cc:1428] Setting key 989360b6cb8f0f7de333f1543981900f
Sep 25 10:26:48 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332007, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:26:48.077234 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.059582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37889 (local address 127.12.248.126:35335)
0925 10:26:48.059759 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.059764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.060881 (+ 1117us) server_negotiation.cc:407] Connection header received
0925 10:26:48.062971 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.062976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.063079 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.063205 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:48.064718 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.065597 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.067574 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.068031 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.068130 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.072054 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:48.072059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:48.072062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:48.072094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:48.074600 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.075284 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.075288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.075290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.075410 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.075781 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.075784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.075786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.076061 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:48.076175 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.076822 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.076989 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:26:48.078269 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.060018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:35335 (local address 127.12.248.67:37889)
0925 10:26:48.060723 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.060763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.062693 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.063455 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.063464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.063969 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:26:48.064542 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.064559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.065752 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.065758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.067429 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.067445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.068254 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.068261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.068551 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.069212 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:26:48.069241 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:26:48.071866 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:26:48.074732 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:48.074739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:48.074752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:48.075177 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:48.075503 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:26:48.075507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:26:48.075509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:26:48.075689 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:26:48.076161 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:26:48.076167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:26:48.076509 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.076741 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.077121 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260925 10:26:48.079936 24122 heartbeater.cc:347] Connected to a master server at 127.12.248.126:35335
I20260925 10:26:48.080302 24122 heartbeater.cc:464] Registering TS with master...
I20260925 10:26:48.081195 24122 heartbeater.cc:511] Master 127.12.248.126:35335 requested a full tablet report, sending...
I20260925 10:26:48.082856 23664 ts_manager.cc:194] Registered new tserver with Master: 8c63175223f74fcc951d463af794f581 (127.12.248.67:44429)
I20260925 10:26:48.083963 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:26:48.084003 23664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:37889
I20260925 10:26:48.108094 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.089941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.12.248.126:35335)
0925 10:26:48.090300 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.090308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.090823 (+ 515us) server_negotiation.cc:407] Connection header received
0925 10:26:48.090992 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.091000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.091113 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.091229 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:48.092844 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.094276 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.096593 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.097284 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.097442 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.101040 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:48.101046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:48.101055 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:48.101091 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:48.104886 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.105912 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.105919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.105925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.106124 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.106573 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.106576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.106578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.106913 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:48.107204 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.107601 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.107790 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 10:26:48.114916 23664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:26:48.122671 23664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:26:48.159222 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.135967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.12.248.67:44429)
0925 10:26:48.150443 (+ 14476us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.150448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.150474 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:26:48.150545 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.150550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.150702 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.151746 (+ 1044us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:48.152039 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.153101 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.156286 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.157719 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.157882 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.158137 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.158688 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.158874 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14332,"thread_start_us":14192,"threads_started":1}
I20260925 10:26:48.159392 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.135311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:44429 (local address 127.0.0.1:58356)
0925 10:26:48.149849 (+ 14538us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.149872 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.149966 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.150976 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.150981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.151013 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:48.151538 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.151550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.153242 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.153249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.156117 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.156132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.157964 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.157970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.158398 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.158490 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.158627 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.158853 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14416,"mutex_wait_us":96,"thread_start_us":11459,"threads_started":1}
I20260925 10:26:48.159444 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.135143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34639 (local address 127.0.0.1:51864)
0925 10:26:48.148553 (+ 13410us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.148610 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.148878 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.150571 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.150584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.150630 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:48.151212 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.151229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.153758 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.153766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.156298 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.156313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.158034 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.158040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.158448 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.158497 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.158627 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.158853 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11828,"mutex_wait_us":203,"thread_start_us":18307,"threads_started":1}
I20260925 10:26:48.162364 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.135393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.12.248.65:34639)
0925 10:26:48.147532 (+ 12139us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.147547 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.149133 (+ 1586us) server_negotiation.cc:407] Connection header received
0925 10:26:48.149264 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.149276 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.149648 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.149883 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:48.151530 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.153634 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.156480 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.157755 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.158258 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.158485 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.161832 (+ 3347us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.162026 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11956,"thread_start_us":11802,"threads_started":1}
I20260925 10:26:48.162549 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.135598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:38263 (local address 127.0.0.1:59798)
0925 10:26:48.153770 (+ 18172us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.153788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.153924 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.154523 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.154530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.154563 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:48.155336 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.155349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.157212 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.157218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.159484 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.159497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.161697 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.161704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.161904 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.161938 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.162027 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.162159 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18019,"mutex_wait_us":7,"thread_start_us":14001,"threads_started":1}
I20260925 10:26:48.162770 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.136192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.12.248.66:38263)
0925 10:26:48.149326 (+ 13134us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.149339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.153933 (+ 4594us) server_negotiation.cc:407] Connection header received
0925 10:26:48.154059 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.154066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.154306 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.154509 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:48.155567 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.157054 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.159688 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.161161 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.161279 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.161517 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.162076 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.162299 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12950,"thread_start_us":12805,"threads_started":1}
I20260925 10:26:48.163944 24057 tablet_service.cc:1467] Processing CreateTablet for tablet 183b7c13746d4a6baa602ae3be1053cd (DEFAULT_TABLE table=test-table [id=0cfde1bf639744ab80fdc05f968b18de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:48.165664 24057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183b7c13746d4a6baa602ae3be1053cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:48.166253 23789 tablet_service.cc:1467] Processing CreateTablet for tablet 183b7c13746d4a6baa602ae3be1053cd (DEFAULT_TABLE table=test-table [id=0cfde1bf639744ab80fdc05f968b18de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:48.166253 23923 tablet_service.cc:1467] Processing CreateTablet for tablet 183b7c13746d4a6baa602ae3be1053cd (DEFAULT_TABLE table=test-table [id=0cfde1bf639744ab80fdc05f968b18de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:26:48.167605 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183b7c13746d4a6baa602ae3be1053cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:48.167608 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183b7c13746d4a6baa602ae3be1053cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:26:48.177127 24140 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Bootstrap starting.
I20260925 10:26:48.181734 24140 tablet_bootstrap.cc:654] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:48.182212 24142 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Bootstrap starting.
I20260925 10:26:48.183240 24140 log.cc:824] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:48.183935 24141 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Bootstrap starting.
I20260925 10:26:48.186000 24142 tablet_bootstrap.cc:654] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:48.187292 24142 log.cc:824] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:48.187542 24140 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: No bootstrap required, opened a new log
I20260925 10:26:48.187577 24141 tablet_bootstrap.cc:654] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Neither blocks nor log segments found. Creating new log.
I20260925 10:26:48.188328 24140 ts_tablet_manager.cc:1397] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 10:26:48.188827 24141 log.cc:824] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:26:48.190557 24142 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: No bootstrap required, opened a new log
I20260925 10:26:48.190851 24142 ts_tablet_manager.cc:1397] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:26:48.192091 24141 tablet_bootstrap.cc:492] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: No bootstrap required, opened a new log
I20260925 10:26:48.192396 24141 ts_tablet_manager.cc:1397] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 10:26:48.199635 24140 raft_consensus.cc:348] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.200103 24140 raft_consensus.cc:374] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:48.200259 24140 raft_consensus.cc:729] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c63175223f74fcc951d463af794f581, State: Initialized, Role: FOLLOWER
I20260925 10:26:48.200145 24141 raft_consensus.cc:348] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.200583 24141 raft_consensus.cc:374] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:48.200696 24141 raft_consensus.cc:729] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2626f5b3ba6c494f9b9579641ef8b1cc, State: Initialized, Role: FOLLOWER
I20260925 10:26:48.201001 24140 consensus_queue.cc:260] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.201282 24141 consensus_queue.cc:260] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.201458 24142 raft_consensus.cc:348] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.201959 24142 raft_consensus.cc:374] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:26:48.202224 24122 heartbeater.cc:503] Master 127.12.248.126:35335 was elected leader, sending a full tablet report...
I20260925 10:26:48.202877 24140 ts_tablet_manager.cc:1428] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260925 10:26:48.203635 24142 raft_consensus.cc:729] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153337bc83c24b6fb9d862a817cb34de, State: Initialized, Role: FOLLOWER
I20260925 10:26:48.203709 23988 heartbeater.cc:503] Master 127.12.248.126:35335 was elected leader, sending a full tablet report...
I20260925 10:26:48.204448 24141 ts_tablet_manager.cc:1428] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 10:26:48.204281 24142 consensus_queue.cc:260] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.206100 24142 ts_tablet_manager.cc:1428] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Time spent starting tablet: real 0.015s user 0.002s sys 0.010s
W20260925 10:26:48.273367 23855 tablet.cc:2369] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:26:48.299681 23989 tablet.cc:2369] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:26:48.308812 24123 tablet.cc:2369] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:26:48.494621 24146 raft_consensus.cc:482] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:26:48.494901 24146 raft_consensus.cc:504] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.496614 24146 leader_election.cc:288] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153337bc83c24b6fb9d862a817cb34de (127.12.248.65:34639), 2626f5b3ba6c494f9b9579641ef8b1cc (127.12.248.66:38263)
I20260925 10:26:48.503103 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.497033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:34639 (local address 127.12.248.67:40657)
0925 10:26:48.497594 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.497616 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.497778 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.498163 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.498168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.498195 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:48.498439 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.498454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.499708 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.499713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.501104 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.501116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.502385 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.502392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.502579 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.502620 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.502721 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.502822 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366}
I20260925 10:26:48.503154 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.497127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:40657 (local address 127.12.248.65:34639)
0925 10:26:48.497372 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.497380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.497739 (+ 359us) server_negotiation.cc:407] Connection header received
0925 10:26:48.497957 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.497962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.498038 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.498131 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:48.498592 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.499564 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.501268 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.502231 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.502360 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.502410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.502757 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.502907 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:26:48.503340 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.497093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:37855 (local address 127.12.248.66:38263)
0925 10:26:48.497397 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.497426 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.497984 (+ 558us) server_negotiation.cc:407] Connection header received
0925 10:26:48.498166 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.498171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.498242 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.498331 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:26:48.499010 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.499720 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.501407 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.502444 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.502591 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.502656 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.502976 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.503095 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 10:26:48.503465 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.497157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:38263 (local address 127.12.248.67:37855)
0925 10:26:48.497871 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 10:26:48.497890 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:26:48.498018 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:26:48.498408 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:26:48.498413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:26:48.498441 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:26:48.498856 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.498868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.499863 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.499867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.501252 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:26:48.501265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.502630 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.502634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:26:48.502808 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.502839 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:26:48.502929 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:26:48.503053 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":371,"threads_started":1}
I20260925 10:26:48.504298 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183b7c13746d4a6baa602ae3be1053cd" candidate_uuid: "8c63175223f74fcc951d463af794f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153337bc83c24b6fb9d862a817cb34de" is_pre_election: true
I20260925 10:26:48.504421 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183b7c13746d4a6baa602ae3be1053cd" candidate_uuid: "8c63175223f74fcc951d463af794f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" is_pre_election: true
I20260925 10:26:48.504916 23809 raft_consensus.cc:2454] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c63175223f74fcc951d463af794f581 in term 0.
I20260925 10:26:48.504916 23943 raft_consensus.cc:2454] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c63175223f74fcc951d463af794f581 in term 0.
I20260925 10:26:48.505695 24012 leader_election.cc:302] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153337bc83c24b6fb9d862a817cb34de, 8c63175223f74fcc951d463af794f581; no voters:
I20260925 10:26:48.506158 24146 raft_consensus.cc:2781] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:26:48.506306 24146 raft_consensus.cc:482] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:26:48.506440 24146 raft_consensus.cc:3037] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:48.508599 24146 raft_consensus.cc:504] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.509200 24146 leader_election.cc:288] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [CANDIDATE]: Term 1 election: Requested vote from peers 153337bc83c24b6fb9d862a817cb34de (127.12.248.65:34639), 2626f5b3ba6c494f9b9579641ef8b1cc (127.12.248.66:38263)
I20260925 10:26:48.509748 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183b7c13746d4a6baa602ae3be1053cd" candidate_uuid: "8c63175223f74fcc951d463af794f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153337bc83c24b6fb9d862a817cb34de"
I20260925 10:26:48.509747 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183b7c13746d4a6baa602ae3be1053cd" candidate_uuid: "8c63175223f74fcc951d463af794f581" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc"
I20260925 10:26:48.509965 23809 raft_consensus.cc:3037] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:48.509965 23943 raft_consensus.cc:3037] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:26:48.513208 23943 raft_consensus.cc:2454] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c63175223f74fcc951d463af794f581 in term 1.
I20260925 10:26:48.513208 23809 raft_consensus.cc:2454] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c63175223f74fcc951d463af794f581 in term 1.
I20260925 10:26:48.513747 24011 leader_election.cc:302] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2626f5b3ba6c494f9b9579641ef8b1cc, 8c63175223f74fcc951d463af794f581; no voters:
I20260925 10:26:48.514074 24146 raft_consensus.cc:2781] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:26:48.514534 24146 raft_consensus.cc:686] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 1 LEADER]: Becoming Leader. State: Replica: 8c63175223f74fcc951d463af794f581, State: Running, Role: LEADER
I20260925 10:26:48.515059 24146 consensus_queue.cc:237] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } }
I20260925 10:26:48.519546 23665 catalog_manager.cc:5495] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c63175223f74fcc951d463af794f581 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "8c63175223f74fcc951d463af794f581" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c63175223f74fcc951d463af794f581" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 44429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 } health_report { overall_health: UNKNOWN } } }
I20260925 10:26:48.614733 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.608938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.12.248.67:44429)
0925 10:26:48.609171 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.609179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.609548 (+ 369us) server_negotiation.cc:407] Connection header received
0925 10:26:48.609808 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.609812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.609890 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.610047 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:26:48.610463 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.611191 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.612573 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.612965 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.613116 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.613611 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:26:48.614124 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:26:48.614294 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.614394 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.614471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:26:48.623116 23809 raft_consensus.cc:1261] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 1 FOLLOWER]: Refusing update from remote peer 8c63175223f74fcc951d463af794f581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:48.623188 23943 raft_consensus.cc:1261] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 1 FOLLOWER]: Refusing update from remote peer 8c63175223f74fcc951d463af794f581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:26:48.624096 24150 consensus_queue.cc:1035] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153337bc83c24b6fb9d862a817cb34de" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:48.624511 24146 consensus_queue.cc:1035] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2626f5b3ba6c494f9b9579641ef8b1cc" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:26:48.636703 24154 mvcc.cc:205] Tried to move back new op lower bound from 7333199907310436352 to 7333199906881339392. Current Snapshot: MvccSnapshot[applied={T|T < 7333199907310436352}]
I20260925 10:26:48.650327 23665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:48.650612 23665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:26:48.654579 23665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6c27fcd392042698b1bac3fff8370df: Sending DeleteTablet for 3 replicas of tablet 183b7c13746d4a6baa602ae3be1053cd
I20260925 10:26:48.655870 24057 tablet_service.cc:1514] Processing DeleteTablet for tablet 183b7c13746d4a6baa602ae3be1053cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:58356
I20260925 10:26:48.655870 23789 tablet_service.cc:1514] Processing DeleteTablet for tablet 183b7c13746d4a6baa602ae3be1053cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:51864
I20260925 10:26:48.656083 23923 tablet_service.cc:1514] Processing DeleteTablet for tablet 183b7c13746d4a6baa602ae3be1053cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:26:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59798
I20260925 10:26:48.658526 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23726
I20260925 10:26:48.688756 24165 tablet_replica.cc:330] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: stopping tablet replica
I20260925 10:26:48.689478 24165 raft_consensus.cc:2229] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:26:48.691025 24165 raft_consensus.cc:2258] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:48.693620 24165 ts_tablet_manager.cc:1905] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:48.694907 24164 tablet_replica.cc:330] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: stopping tablet replica
I20260925 10:26:48.697955 24166 tablet_replica.cc:330] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: stopping tablet replica
I20260925 10:26:48.698427 24164 raft_consensus.cc:2229] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:48.698508 24166 raft_consensus.cc:2229] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:26:48.699014 24164 raft_consensus.cc:2258] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:26:48.699249 24166 raft_consensus.cc:2258] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 10:26:48 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:26:48.701530 24164 ts_tablet_manager.cc:1905] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:48.701637 24165 ts_tablet_manager.cc:1918] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:48.701860 24165 log.cc:1196] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/183b7c13746d4a6baa602ae3be1053cd
I20260925 10:26:48.702522 24165 ts_tablet_manager.cc:1939] T 183b7c13746d4a6baa602ae3be1053cd P 8c63175223f74fcc951d463af794f581: Deleting consensus metadata
I20260925 10:26:48.703183 24166 ts_tablet_manager.cc:1905] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:26:48.704437 23649 catalog_manager.cc:4841] TS 8c63175223f74fcc951d463af794f581 (127.12.248.67:44429): tablet 183b7c13746d4a6baa602ae3be1053cd (table test-table [id=0cfde1bf639744ab80fdc05f968b18de]) successfully deleted
I20260925 10:26:48.710373 24164 ts_tablet_manager.cc:1918] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:48.710604 24164 log.cc:1196] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/183b7c13746d4a6baa602ae3be1053cd
I20260925 10:26:48.711272 24164 ts_tablet_manager.cc:1939] T 183b7c13746d4a6baa602ae3be1053cd P 153337bc83c24b6fb9d862a817cb34de: Deleting consensus metadata
I20260925 10:26:48.711638 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.660261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.12.248.65:34639)
0925 10:26:48.660484 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.660493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.684165 (+ 23672us) server_negotiation.cc:407] Connection header received
0925 10:26:48.684881 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.684891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.685027 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.685124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:48.688239 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.690373 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.692794 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.693429 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.693629 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.703502 (+ 9873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:48.703517 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:48.703539 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:48.703593 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:48.707669 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.708642 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.708653 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.708656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.708837 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.709385 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.709390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.709393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.709901 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:48.710209 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.710858 (+ 649us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.711131 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:26:48.712451 24166 ts_tablet_manager.cc:1918] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:26:48.712678 24166 log.cc:1196] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/183b7c13746d4a6baa602ae3be1053cd
I20260925 10:26:48.713357 23849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:48.713459 24166 ts_tablet_manager.cc:1939] T 183b7c13746d4a6baa602ae3be1053cd P 2626f5b3ba6c494f9b9579641ef8b1cc: Deleting consensus metadata
I20260925 10:26:48.713470 23651 catalog_manager.cc:4841] TS 153337bc83c24b6fb9d862a817cb34de (127.12.248.65:34639): tablet 183b7c13746d4a6baa602ae3be1053cd (table test-table [id=0cfde1bf639744ab80fdc05f968b18de]) successfully deleted
I20260925 10:26:48.716185 23650 catalog_manager.cc:4841] TS 2626f5b3ba6c494f9b9579641ef8b1cc (127.12.248.66:38263): tablet 183b7c13746d4a6baa602ae3be1053cd (table test-table [id=0cfde1bf639744ab80fdc05f968b18de]) successfully deleted
I20260925 10:26:48.866746 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23726
I20260925 10:26:48.884948 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23860
Sep 25 10:26:48 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:26:48.902810 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:48.885827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.12.248.66:38263)
0925 10:26:48.886028 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:48.886032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:48.886128 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:26:48.886332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:48.886336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:48.886448 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:48.886553 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:48.888170 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.888922 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.890859 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:48.891352 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:48.891453 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:48.896469 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:48.896484 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:48.896501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:48.896537 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:48.899838 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.900528 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.900538 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.900541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.900671 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:48.901048 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:48.901053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:48.901057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:48.901517 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:48.901809 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:48.902237 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 10:26:48.902463 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:26:48.903458 23983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:48.996667 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23860
I20260925 10:26:49.020396 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23994
Sep 25 10:26:49 dist-test-slave-f9rl krb5kdc[23620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:26:49.039237 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:26:49.021245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.12.248.67:44429)
0925 10:26:49.021427 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:26:49.021432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:26:49.021516 (+ 84us) server_negotiation.cc:407] Connection header received
0925 10:26:49.021718 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:26:49.021722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:26:49.021786 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:26:49.021862 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:26:49.023620 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:49.024416 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:49.026290 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:26:49.026823 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:26:49.026944 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:26:49.032750 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:26:49.032764 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:26:49.032779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:26:49.032838 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:26:49.035977 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:49.036841 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:49.036848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:49.036851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:49.037003 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:26:49.037530 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:26:49.037533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:26:49.037535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:26:49.037910 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:26:49.038185 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 10:26:49.038626 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 10:26:49.038810 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:26:49.040091 24117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:49.167122 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23994
I20260925 10:26:49.187235 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 23633
I20260925 10:26:49.188522 23695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:26:49.298974 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 23633
2026-09-25T10:26:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4692 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:26:49 dist-test-slave-f9rl krb5kdc[24174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:26:49 dist-test-slave-f9rl krb5kdc[24174](info): set up 2 sockets
Sep 25 10:26:49 dist-test-slave-f9rl krb5kdc[24174](info): commencing operation
krb5kdc: starting...
W20260925 10:26:50.503947 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:26:50 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:26:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangeradmin_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangeradmin_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangerlookup_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangerlookup_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
I20260925 10:26:50.844509 13281 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZKvIMz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/postgres -l logfile start
2026-09-25 10:26:53.556 UTC [24204] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:26:53.557 UTC [24204] LOG: listening on IPv4 address "127.12.248.84", port 56779
2026-09-25 10:26:53.560 UTC [24204] LOG: listening on Unix socket "/tmp/.s.PGSQL.56779"
2026-09-25 10:26:53.582 UTC [24207] LOG: database system was shut down at 2026-09-25 10:26:51 UTC
2026-09-25 10:26:53.586 UTC [24204] LOG: database system is ready to accept connections
I20260925 10:26:54.551450 13281 mini_postgres.cc:96] Postgres bound to 56779
2026-09-25 10:26:54.569 UTC [24215] FATAL: database "slave" does not exist
127.12.248.84:56779 - accepting connections
I20260925 10:26:54.570013 13281 mini_ranger.cc:161] Starting Ranger...
I20260925 10:26:54.596625 13281 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:26:55.138330 13281 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:26:55.138494 13281 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin
2026-09-25 10:26:55,445 [I] DB FLAVOR :POSTGRES
2026-09-25 10:26:55,445 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:26:55,445 [I] Checking connection..
2026-09-25 10:26:55,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:26:55,714 [I] Checking connection passed.
2026-09-25 10:26:55,715 [I] --------- Verifying version history table ---------
2026-09-25 10:26:55,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:26:55,997 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:26:55,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:26:56,271 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:26:56,272 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:26:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:26:56.537 UTC [24279] WARNING: there is no transaction in progress
2026-09-25 10:26:56,545 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:26:56,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:26:56,843 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:26:56,843 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:26:56,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:26:57,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:26:57,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rl.c.gcp-upstream.internal','N') ;"
2026-09-25 10:26:57,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:26:57,895 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:26:57,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:26:58,158 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:26:58,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:26:58,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:26:58,698 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:26:58,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:26:59.798 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.817 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.836 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.855 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.874 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.895 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.958 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.965 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.981 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:26:59.992 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.003 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.018 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.029 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.036 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.043 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00.051 UTC [24387] WARNING: there is no transaction in progress
2026-09-25 10:27:00,736 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:27:00,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:27:00,992 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:27:00,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:27:01,242 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:27:01,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:27:01,511 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:27:01,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:27:01,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:27:01,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rl.c.gcp-upstream.internal';"
2026-09-25 10:27:02,257 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:27:02,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:27:02,525 [I] DB_PATCHES have already been applied
I20260925 10:27:02.531986 13281 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZKvIMz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:27:02.532076 13281 mini_ranger.cc:193] Using host: 127.12.248.84
I20260925 10:27:02.547114 13281 mini_ranger.cc:227] Ranger admin URL: http://127.12.248.84:56461
Sep 25, 2026 10:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:27:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:27:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:27:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:27:03.680539 13281 mini_ranger.cc:160] Time spent starting Ranger: real 9.111s user 0.002s sys 0.039s
Sep 25, 2026 10:27:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:27:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:27:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:27:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:27:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:27:25.187259 13281 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangerkms_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/rangerkms_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.248.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.248.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:27:25 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:27:25.352121 13281 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:27:25.375172 13281 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:27:26.218469 13281 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:27:26.218649 13281 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms
I20260925 10:27:26.218777 13281 mini_ranger_kms.cc:227] Using postgres at 127.12.248.84:56779
2026-09-25 10:27:26,655 [I] DB FLAVOR :POSTGRES
2026-09-25 10:27:26,655 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:27:26,655 [I] Checking connection
2026-09-25 10:27:26,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:27:26,935 [I] connection success
2026-09-25 10:27:26,936 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:27:26,936 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:27:26,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:27:27,177 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:27:27,178 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:27:27,178 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:27:27,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:56779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:27:27,465 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:27:27.935165 13281 mini_ranger_kms.cc:310] Created kms service
I20260925 10:27:28.132818 13281 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:27:28.180315 13281 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:27:28 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:28.583240 13281 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:27:28.583582 13281 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:27:28.583642 13281 mini_ranger_kms.cc:240] Using host: 127.12.248.84
I20260925 10:27:28.601810 13281 mini_ranger_kms.cc:276] Ranger KMS PID: 24567
I20260925 10:27:28.602041 13281 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.248.84:37061
Sep 25, 2026 10:27:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:27:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:27:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:27:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:27:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:27:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:27:30.935616 13281 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.584s user 0.146s sys 0.310s
I20260925 10:27:30.935891 13281 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:27:30.936091 13281 mini_ranger_kms.cc:401] 127.12.248.84:37061/kms/v1/keys
Sep 25 10:27:31 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:27:31 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:27:31 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:27:34 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:27:34.212795 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44457
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44457
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:37061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:27:34.627579 24618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:27:34.628027 24618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:27:34.628257 24618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:27:34.637028 24618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:27:34.637142 24618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:27:34.637194 24618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:27:34.637257 24618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:27:34.637305 24618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:27:34.649955 24618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.248.84:37061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44457
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:27:34.651432 24618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:27:34.652818 24618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:27:34.666970 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:34.667712 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:34.667768 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:27:34.668627 24618 server_base.cc:1034] running on GCE node
I20260925 10:27:34.669410 24618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:27:34.670996 24618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:27:34.672228 24618 hybrid_clock.cc:648] HybridClock initialized: now 1790332054672187 us; error 57 us; skew 500 ppm
I20260925 10:27:34.672587 24618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:27:34 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:27:34.681020 24618 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:27:34.683189 24618 webserver.cc:460] Webserver started at http://127.12.248.126:34487/ using document root <none> and password file <none>
I20260925 10:27:34.684058 24618 fs_manager.cc:359] Metadata directory not provided
I20260925 10:27:34.684187 24618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:27:34.684540 24618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:27:34 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:34.737085 24618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "35c169822b3c40a9891e3c4ff37eaee0"
format_stamp: "Formatted at 2026-09-25 10:27:34 on dist-test-slave-f9rl"
server_key: "1735c424cad8d7f7e2f9f14a992909ba"
server_key_iv: "e2fc4583b0a81f6bf33955bac036610a"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:34.738039 24618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "35c169822b3c40a9891e3c4ff37eaee0"
format_stamp: "Formatted at 2026-09-25 10:27:34 on dist-test-slave-f9rl"
server_key: "1735c424cad8d7f7e2f9f14a992909ba"
server_key_iv: "e2fc4583b0a81f6bf33955bac036610a"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:34.744221 24618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 10:27:34.761899 24633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:34.763721 24618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:27:34.763947 24618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "35c169822b3c40a9891e3c4ff37eaee0"
format_stamp: "Formatted at 2026-09-25 10:27:34 on dist-test-slave-f9rl"
server_key: "1735c424cad8d7f7e2f9f14a992909ba"
server_key_iv: "e2fc4583b0a81f6bf33955bac036610a"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:34.764205 24618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:27:34.786067 24618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:27:34.791018 24618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:27:34.791368 24618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:27:34.816519 24618 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:44457
I20260925 10:27:34.816553 24684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:44457 every 8 connection(s)
I20260925 10:27:34.818145 24618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:27:34.823050 24685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:27:34.824334 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 24618
I20260925 10:27:34.824543 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:27:34.825363 13281 external_mini_cluster.cc:1428] Setting key 3d1fee0ee0f2fdddc8d3db60b3032390
I20260925 10:27:34.836221 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: Bootstrap starting.
I20260925 10:27:34.840457 24685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: Neither blocks nor log segments found. Creating new log.
Sep 25 10:27:34 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:27:34.841799 24685 log.cc:824] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:27:34.845173 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: No bootstrap required, opened a new log
I20260925 10:27:34.850703 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:34.827941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.12.248.126:44457)
0925 10:27:34.828990 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:34.829004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:34.829048 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:27:34.831197 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:34.831223 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:34.831644 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:34.831992 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:34.833793 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:34.835144 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:34.836934 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:34.837679 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:34.837822 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:34.842898 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:34.842910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:34.842926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:34.842964 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:34.846527 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:34.847276 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:34.847283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:34.847292 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:34.847460 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:34.847882 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:34.847892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:34.847897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:34.848432 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:34.848765 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:34.849202 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 10:27:34.849652 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":549,"threads_started":1}
I20260925 10:27:34.854246 24685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } }
I20260925 10:27:34.854625 24685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:27:34.854734 24685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c169822b3c40a9891e3c4ff37eaee0, State: Initialized, Role: FOLLOWER
I20260925 10:27:34.855324 24685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } }
I20260925 10:27:34.855578 24685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:27:34.855713 24685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:27:34.855888 24685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:27:34.858232 24685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } }
I20260925 10:27:34.858803 24685 leader_election.cc:302] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35c169822b3c40a9891e3c4ff37eaee0; no voters:
I20260925 10:27:34.859547 24685 leader_election.cc:288] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:27:34.859664 24690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:27:34.860555 24690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [term 1 LEADER]: Becoming Leader. State: Replica: 35c169822b3c40a9891e3c4ff37eaee0, State: Running, Role: LEADER
I20260925 10:27:34.861272 24690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } }
I20260925 10:27:34.862325 24685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:27:34.864530 24692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35c169822b3c40a9891e3c4ff37eaee0. Latest consensus state: current_term: 1 leader_uuid: "35c169822b3c40a9891e3c4ff37eaee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } } }
I20260925 10:27:34.864606 24691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35c169822b3c40a9891e3c4ff37eaee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c169822b3c40a9891e3c4ff37eaee0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44457 } } }
I20260925 10:27:34.865134 24691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:27:34.865123 24692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:27:34.870992 24695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:27:34.877002 24695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:27:34.884959 24685 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:27:34.902079 24695 catalog_manager.cc:1347] Generated new cluster ID: c6de59f7c95b4dd1a01c79c056743a1b
I20260925 10:27:34.902308 24695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:27:34.917400 24695 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:27:34.919497 24695 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:27:34.933405 24695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: Generated new TSK 0
I20260925 10:27:34.934764 24695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:27:35.801605 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:37061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:27:36.308516 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:27:36.308912 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:27:36.309122 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:27:36.321139 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:27:36.321372 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:27:36.321718 24730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:27:36.339661 24730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.248.84:37061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:27:36.341594 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:27:36.343325 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:27:36.359634 24740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:36.361238 24741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:36.361652 24743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:27:36.361729 24730 server_base.cc:1034] running on GCE node
I20260925 10:27:36.362396 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:27:36.363528 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:27:36.364849 24730 hybrid_clock.cc:648] HybridClock initialized: now 1790332056364811 us; error 39 us; skew 500 ppm
I20260925 10:27:36.365247 24730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:27:36.369841 24730 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:27:36.386611 24730 webserver.cc:460] Webserver started at http://127.12.248.65:40393/ using document root <none> and password file <none>
I20260925 10:27:36.387650 24730 fs_manager.cc:359] Metadata directory not provided
I20260925 10:27:36.387835 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:27:36.388274 24730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:36.408541 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "978236a5244e4fb2b860ac852a94076d"
format_stamp: "Formatted at 2026-09-25 10:27:36 on dist-test-slave-f9rl"
server_key: "8c002b3c81146325f63748a55291816f"
server_key_iv: "497586ebfeb9b61c7e81b14956575236"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:36.409994 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "978236a5244e4fb2b860ac852a94076d"
format_stamp: "Formatted at 2026-09-25 10:27:36 on dist-test-slave-f9rl"
server_key: "8c002b3c81146325f63748a55291816f"
server_key_iv: "497586ebfeb9b61c7e81b14956575236"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:36.425228 24730 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.013s sys 0.001s
I20260925 10:27:36.462620 24750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:36.465152 24730 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.015s sys 0.003s
I20260925 10:27:36.465484 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "978236a5244e4fb2b860ac852a94076d"
format_stamp: "Formatted at 2026-09-25 10:27:36 on dist-test-slave-f9rl"
server_key: "8c002b3c81146325f63748a55291816f"
server_key_iv: "497586ebfeb9b61c7e81b14956575236"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:36.465888 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:27:36.550498 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:27:36.556761 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:27:36.557195 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:27:36.564801 24730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:27:36.573664 24730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:27:36.573858 24730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:36.574088 24730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:27:36.574209 24730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:36.701648 24730 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:40649
I20260925 10:27:36.701695 24862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:40649 every 8 connection(s)
I20260925 10:27:36.703359 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:27:36.704818 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 24730
I20260925 10:27:36.705075 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:27:36.706240 13281 external_mini_cluster.cc:1428] Setting key a62a0116ab3e490fdc1d628f78bbab45
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:27:36.751537 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:36.721168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:56433 (local address 127.12.248.126:44457)
0925 10:27:36.730187 (+ 9019us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:36.730192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:36.730211 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:27:36.730265 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:36.730270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:36.730341 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:36.730495 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:36.732184 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:36.732955 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:36.734664 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:36.735092 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:36.735238 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:36.742791 (+ 7553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:36.742802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:36.742809 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:36.742860 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:36.746946 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:36.748216 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:36.748226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:36.748232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:36.748406 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:36.749030 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:36.749038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:36.749044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:36.749471 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:36.749623 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:36.750790 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0925 10:27:36.751069 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8883,"thread_start_us":8787,"threads_started":1}
I20260925 10:27:36.753427 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:36.720909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44457 (local address 127.12.248.65:56433)
0925 10:27:36.721893 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:36.721955 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:36.724426 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:36.730798 (+ 6372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:36.730809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:36.731298 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:27:36.731924 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:36.731944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:36.733129 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:36.733135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:36.734499 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:36.734520 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:36.735339 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:36.735346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:36.735606 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:36.736429 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:27:36.736452 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:27:36.742463 (+ 6011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:27:36.747159 (+ 4696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:36.747173 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:36.747198 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:36.748043 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:36.748538 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:36.748545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:36.748550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:36.748865 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:36.749624 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:27:36.749634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:27:36.750242 (+ 608us) client_negotiation.cc:769] Sending connection context
0925 10:27:36.750670 (+ 428us) client_negotiation.cc:240] Negotiation successful
0925 10:27:36.751392 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":447,"threads_started":1}
I20260925 10:27:36.755772 24863 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44457
I20260925 10:27:36.756085 24863 heartbeater.cc:464] Registering TS with master...
I20260925 10:27:36.756983 24863 heartbeater.cc:511] Master 127.12.248.126:44457 requested a full tablet report, sending...
I20260925 10:27:36.759826 24650 ts_manager.cc:194] Registered new tserver with Master: 978236a5244e4fb2b860ac852a94076d (127.12.248.65:40649)
I20260925 10:27:36.761968 24650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:56433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:27:36.777736 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:37061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:27:36 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
W20260925 10:27:37.325583 24869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:27:37.325866 24869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:27:37.325977 24869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:27:37.333920 24869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:27:37.334153 24869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:27:37.334393 24869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:27:37.345197 24869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.248.84:37061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:27:37.346701 24869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:27:37.348137 24869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:27:37.361722 24883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:37.361693 24884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:37.361912 24886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:27:37.362931 24869 server_base.cc:1034] running on GCE node
I20260925 10:27:37.363648 24869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:27:37.364660 24869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:27:37.365886 24869 hybrid_clock.cc:648] HybridClock initialized: now 1790332057365841 us; error 39 us; skew 500 ppm
I20260925 10:27:37.366225 24869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:27:37 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:27:37.369524 24869 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:27:37.371147 24869 webserver.cc:460] Webserver started at http://127.12.248.66:43503/ using document root <none> and password file <none>
I20260925 10:27:37.371840 24869 fs_manager.cc:359] Metadata directory not provided
I20260925 10:27:37.371975 24869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:27:37.372308 24869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:27:37 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:37.390658 24869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "65171ac073b441a1a76cacb24cccf937"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "b88bca854e8c9e61b91826fe5f25f728"
server_key_iv: "a09bd7677963c65c5452d325032d11ef"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:37.391510 24869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "65171ac073b441a1a76cacb24cccf937"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "b88bca854e8c9e61b91826fe5f25f728"
server_key_iv: "a09bd7677963c65c5452d325032d11ef"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:37.397490 24869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260925 10:27:37.413590 24893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:37.415282 24869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:27:37.415465 24869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "65171ac073b441a1a76cacb24cccf937"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "b88bca854e8c9e61b91826fe5f25f728"
server_key_iv: "a09bd7677963c65c5452d325032d11ef"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:37.415702 24869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:27:37.430363 24869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:27:37.434726 24869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:27:37.435122 24869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:27:37.436070 24869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:27:37.437593 24869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:27:37.437667 24869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:37.437834 24869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:27:37.437887 24869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:37.476388 24869 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:40015
I20260925 10:27:37.476429 25005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:40015 every 8 connection(s)
I20260925 10:27:37.477839 24869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:27:37.478109 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 24869
I20260925 10:27:37.478276 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:27:37.478713 13281 external_mini_cluster.cc:1428] Setting key 92a1e0af64a6b44b93320cd4750fdd02
Sep 25 10:27:37 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:27:37.499230 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:37.480844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:49825 (local address 127.12.248.126:44457)
0925 10:27:37.481506 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:37.481512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:37.482649 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 10:27:37.484523 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:37.484528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:37.484638 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:37.484777 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:37.486341 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:37.487203 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:37.488780 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:37.489253 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:37.489397 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:37.493147 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:37.493152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:37.493155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:37.493192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:37.496002 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:37.496759 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:37.496770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:37.496772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:37.496889 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:37.497345 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:37.497350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:37.497355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:37.497699 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:37.497855 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:37.498628 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 10:27:37.498847 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":358,"threads_started":1}
I20260925 10:27:37.500241 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:37.481695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44457 (local address 127.12.248.66:49825)
0925 10:27:37.482497 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:37.482536 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:37.484248 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:37.485038 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:37.485047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:37.485559 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:27:37.486169 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:37.486186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:37.487377 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:37.487383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:37.488623 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:37.488634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:37.489603 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:37.489615 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:37.489940 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:37.490657 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:27:37.490681 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:27:37.492920 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:27:37.496161 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:37.496170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:37.496188 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:37.496625 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:37.497002 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:37.497007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:37.497011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:37.497220 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:37.497832 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:27:37.497840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:27:37.498247 (+ 407us) client_negotiation.cc:769] Sending connection context
0925 10:27:37.498515 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 10:27:37.499061 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:27:37.501711 25006 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44457
I20260925 10:27:37.502032 25006 heartbeater.cc:464] Registering TS with master...
I20260925 10:27:37.502867 25006 heartbeater.cc:511] Master 127.12.248.126:44457 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:27:37.504454 24650 ts_manager.cc:194] Registered new tserver with Master: 65171ac073b441a1a76cacb24cccf937 (127.12.248.66:40015)
I20260925 10:27:37.505234 24650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:49825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:27:37.532135 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:37061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:27:37.766324 24863 heartbeater.cc:503] Master 127.12.248.126:44457 was elected leader, sending a full tablet report...
W20260925 10:27:37.911288 25012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:27:37.911612 25012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:27:37.911778 25012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:27:37.920310 25012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:27:37.920463 25012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:27:37.920634 25012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:27:37.931473 25012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.248.84:37061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:27:37.933247 25012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:27:37.934696 25012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:27:37.947966 25018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:37.948022 25020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:27:37.948056 25017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:27:37.954156 25012 server_base.cc:1034] running on GCE node
I20260925 10:27:37.954790 25012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:27:37.955806 25012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:27:37.957002 25012 hybrid_clock.cc:648] HybridClock initialized: now 1790332057956938 us; error 54 us; skew 500 ppm
I20260925 10:27:37.957324 25012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:27:37 dist-test-slave-f9rl krb5kdc[24174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:27:37.960402 25012 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:27:37.961900 25012 webserver.cc:460] Webserver started at http://127.12.248.67:43415/ using document root <none> and password file <none>
I20260925 10:27:37.962558 25012 fs_manager.cc:359] Metadata directory not provided
I20260925 10:27:37.962689 25012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:27:37.963110 25012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:27:37 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:27:37.978116 25012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "811dfe326e8f48ae90b3ff9c796dc567"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "1aeadb8e294a7e2bee76653985211876"
server_key_iv: "919f8ed4303e6a0fc4f1bdccbbb2951c"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:37.978968 25012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "811dfe326e8f48ae90b3ff9c796dc567"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "1aeadb8e294a7e2bee76653985211876"
server_key_iv: "919f8ed4303e6a0fc4f1bdccbbb2951c"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:37.984525 25012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 10:27:38.001353 25027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:38.003259 25012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:27:38.003438 25012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "811dfe326e8f48ae90b3ff9c796dc567"
format_stamp: "Formatted at 2026-09-25 10:27:37 on dist-test-slave-f9rl"
server_key: "1aeadb8e294a7e2bee76653985211876"
server_key_iv: "919f8ed4303e6a0fc4f1bdccbbb2951c"
server_key_version: "kuduclusterkey@0"
I20260925 10:27:38.003723 25012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:27:38.027727 25012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:27:38.031607 25012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:27:38.031885 25012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:27:38.032821 25012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:27:38.034281 25012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:27:38.034426 25012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:38.034646 25012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:27:38.034808 25012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:27:38.073805 25012 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:35561
I20260925 10:27:38.073825 25139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:35561 every 8 connection(s)
I20260925 10:27:38.075588 25012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:27:38.085366 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 25012
I20260925 10:27:38.085604 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:27:38.086091 13281 external_mini_cluster.cc:1428] Setting key 30c0f1a403605401c45c4f13af0b325c
Sep 25 10:27:38 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332057, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:27:38.097501 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.078515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:38935 (local address 127.12.248.126:44457)
0925 10:27:38.080646 (+ 2131us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.080654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.080677 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:27:38.082355 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.082363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.082493 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.082617 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:38.084188 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.085147 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.087163 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.087736 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.087873 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.091437 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:38.091444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:38.091449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:38.091493 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:38.094139 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.094887 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.094894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.094897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.095018 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.095484 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.095494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.095496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.095768 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:38.095927 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.096852 (+ 925us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.097083 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1789,"threads_started":1}
I20260925 10:27:38.098528 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.078889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44457 (local address 127.12.248.67:38935)
0925 10:27:38.079836 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.079892 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.082073 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.082890 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.082898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.083427 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:27:38.084019 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.084036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.085362 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.085366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.086963 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.086983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.087979 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.087988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.088259 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.088962 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:27:38.088989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:27:38.091205 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:27:38.094288 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:38.094297 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:38.094315 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:38.094747 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:38.095132 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:27:38.095137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:27:38.095141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:27:38.095358 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:27:38.095941 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:27:38.095952 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:27:38.096448 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.096758 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.097304 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":406,"threads_started":1}
I20260925 10:27:38.100126 25140 heartbeater.cc:347] Connected to a master server at 127.12.248.126:44457
I20260925 10:27:38.100482 25140 heartbeater.cc:464] Registering TS with master...
I20260925 10:27:38.101274 25140 heartbeater.cc:511] Master 127.12.248.126:44457 requested a full tablet report, sending...
I20260925 10:27:38.102861 24650 ts_manager.cc:194] Registered new tserver with Master: 811dfe326e8f48ae90b3ff9c796dc567 (127.12.248.67:35561)
I20260925 10:27:38.103729 24650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:38935
I20260925 10:27:38.114321 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:27:38.132316 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.119076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.12.248.126:44457)
0925 10:27:38.119318 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.119328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.119695 (+ 367us) server_negotiation.cc:407] Connection header received
0925 10:27:38.119904 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.119909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.119994 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.120100 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:38.121979 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.122717 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.124173 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.124603 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.124747 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.127182 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:38.127187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:38.127190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:38.127222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:38.129704 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.130342 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.130349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.130353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.130484 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.130866 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.130872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.130876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.131153 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:38.131409 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.131755 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.131997 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:27:38.136370 24650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:27:38.143923 24650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:27:38.180145 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.156730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35561 (local address 127.0.0.1:58474)
0925 10:27:38.168482 (+ 11752us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.168547 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.170348 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.172100 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.172111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.172148 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:27:38.172659 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.172678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.175062 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.175070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.177314 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.177325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.179003 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.179009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.179202 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.179259 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.179426 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.179678 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11384,"thread_start_us":11208,"threads_started":1}
I20260925 10:27:38.180845 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.157670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:40015 (local address 127.0.0.1:52764)
0925 10:27:38.171597 (+ 13927us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.171616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.171722 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.174040 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.174046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.174069 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:27:38.174483 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.174494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.175972 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.175980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.177941 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.177956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.179978 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.179984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.180165 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.180219 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.180311 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.180440 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13812,"mutex_wait_us":1,"thread_start_us":11043,"threads_started":1}
I20260925 10:27:38.181214 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.157076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.12.248.67:35561)
0925 10:27:38.171194 (+ 14118us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.171203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.171238 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:27:38.171453 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.171462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.171757 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.172859 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:27:38.173348 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.174884 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.177515 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.178801 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.178931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.179171 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.179485 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.179656 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13953,"thread_start_us":13690,"threads_started":1}
I20260925 10:27:38.181618 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.157748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.12.248.66:40015)
0925 10:27:38.173161 (+ 15413us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.173171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.173203 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:27:38.173301 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.173310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.173587 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.173774 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:27:38.174686 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.175842 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.178125 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.179462 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.179596 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.179807 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.181069 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.181243 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15246,"thread_start_us":15145,"threads_started":1}
I20260925 10:27:38.182036 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.171218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:40649 (local address 127.0.0.1:39314)
0925 10:27:38.173455 (+ 2237us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.173477 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.173601 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.174473 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.174478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.174501 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:27:38.174953 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.174964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.176903 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.176909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.178645 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.178655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.181129 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.181139 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.181329 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.181360 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.181489 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.181641 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2091,"thread_start_us":1504,"threads_started":1}
I20260925 10:27:38.182139 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.169574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.12.248.65:40649)
0925 10:27:38.172607 (+ 3033us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.172619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.173572 (+ 953us) server_negotiation.cc:407] Connection header received
0925 10:27:38.173742 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.173750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.173969 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.174187 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:27:38.175142 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.176745 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.178784 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.180528 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.180672 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.180919 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.181535 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.181781 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2827,"thread_start_us":2646,"threads_started":1}
I20260925 10:27:38.184909 25075 tablet_service.cc:1467] Processing CreateTablet for tablet cea333e7775441568814ae4a91a3e764 (DEFAULT_TABLE table=test-table [id=3c16f34938884c7483bf4c91a0615b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:27:38.185130 24798 tablet_service.cc:1467] Processing CreateTablet for tablet cea333e7775441568814ae4a91a3e764 (DEFAULT_TABLE table=test-table [id=3c16f34938884c7483bf4c91a0615b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:27:38.186350 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea333e7775441568814ae4a91a3e764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:27:38.186605 24798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea333e7775441568814ae4a91a3e764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:27:38.188079 24941 tablet_service.cc:1467] Processing CreateTablet for tablet cea333e7775441568814ae4a91a3e764 (DEFAULT_TABLE table=test-table [id=3c16f34938884c7483bf4c91a0615b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:27:38.189487 24941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cea333e7775441568814ae4a91a3e764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:27:38.198000 25159 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Bootstrap starting.
I20260925 10:27:38.200996 25159 tablet_bootstrap.cc:654] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Neither blocks nor log segments found. Creating new log.
I20260925 10:27:38.201854 25160 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Bootstrap starting.
I20260925 10:27:38.202320 25159 log.cc:824] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Log is configured to *not* fsync() on all Append() calls
I20260925 10:27:38.204855 25162 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Bootstrap starting.
I20260925 10:27:38.205555 25159 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: No bootstrap required, opened a new log
I20260925 10:27:38.205811 25160 tablet_bootstrap.cc:654] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Neither blocks nor log segments found. Creating new log.
I20260925 10:27:38.205930 25159 ts_tablet_manager.cc:1397] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 10:27:38.207214 25160 log.cc:824] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:27:38.208397 25162 tablet_bootstrap.cc:654] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Neither blocks nor log segments found. Creating new log.
I20260925 10:27:38.209700 25162 log.cc:824] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Log is configured to *not* fsync() on all Append() calls
I20260925 10:27:38.212167 25160 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: No bootstrap required, opened a new log
I20260925 10:27:38.212673 25160 ts_tablet_manager.cc:1397] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260925 10:27:38.214612 25162 tablet_bootstrap.cc:492] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: No bootstrap required, opened a new log
I20260925 10:27:38.214881 25162 ts_tablet_manager.cc:1397] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 10:27:38.218727 25159 raft_consensus.cc:348] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.219271 25159 raft_consensus.cc:374] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:27:38.219413 25159 raft_consensus.cc:729] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811dfe326e8f48ae90b3ff9c796dc567, State: Initialized, Role: FOLLOWER
I20260925 10:27:38.220149 25159 consensus_queue.cc:260] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.221941 25140 heartbeater.cc:503] Master 127.12.248.126:44457 was elected leader, sending a full tablet report...
I20260925 10:27:38.222612 25159 ts_tablet_manager.cc:1428] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260925 10:27:38.224017 25162 raft_consensus.cc:348] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.224465 25162 raft_consensus.cc:374] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:27:38.224565 25162 raft_consensus.cc:729] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65171ac073b441a1a76cacb24cccf937, State: Initialized, Role: FOLLOWER
I20260925 10:27:38.224786 25160 raft_consensus.cc:348] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.225201 25160 raft_consensus.cc:374] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:27:38.225200 25162 consensus_queue.cc:260] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.225325 25160 raft_consensus.cc:729] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978236a5244e4fb2b860ac852a94076d, State: Initialized, Role: FOLLOWER
I20260925 10:27:38.226037 25160 consensus_queue.cc:260] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.227079 25162 ts_tablet_manager.cc:1428] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260925 10:27:38.227784 25160 ts_tablet_manager.cc:1428] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260925 10:27:38.228334 25006 heartbeater.cc:503] Master 127.12.248.126:44457 was elected leader, sending a full tablet report...
W20260925 10:27:38.230360 25007 tablet.cc:2369] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:27:38.256080 25166 raft_consensus.cc:482] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:27:38.256404 25166 raft_consensus.cc:504] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.258186 25166 leader_election.cc:288] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 811dfe326e8f48ae90b3ff9c796dc567 (127.12.248.67:35561), 978236a5244e4fb2b860ac852a94076d (127.12.248.65:40649)
I20260925 10:27:38.265669 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.258561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:35561 (local address 127.12.248.66:51517)
0925 10:27:38.259205 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.259222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.259390 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.259860 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.259867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.259901 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:27:38.260376 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.260402 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.261651 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.261656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.263464 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.263476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.264773 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.264780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.264964 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.265001 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.265103 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.265227 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":20,"spinlock_wait_cycles":28800,"thread_start_us":370,"threads_started":1}
I20260925 10:27:38.265787 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.258629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:51517 (local address 127.12.248.67:35561)
0925 10:27:38.258876 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.258886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.259317 (+ 431us) server_negotiation.cc:407] Connection header received
0925 10:27:38.259551 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.259559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.259648 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.259744 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:27:38.260531 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.261528 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.263635 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.264547 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.264672 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.264733 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.265158 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.265346 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:27:38.265985 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.259629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:52683 (local address 127.12.248.65:40649)
0925 10:27:38.259845 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.259850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.260070 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:27:38.260239 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.260244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.260330 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.260433 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:27:38.261063 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.261975 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.263939 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.264844 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.265010 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.265096 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.265546 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.265703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:27:38.266721 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.259337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:40649 (local address 127.12.248.66:52683)
0925 10:27:38.259963 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 10:27:38.259981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:27:38.260097 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:27:38.260475 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:27:38.260482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:27:38.260507 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:27:38.260912 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.260924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.262117 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.262120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.263766 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:27:38.263780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.265117 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.265123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:27:38.265316 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.265352 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:27:38.265506 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 10:27:38.266442 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":24,"spinlock_wait_cycles":896,"thread_start_us":313,"threads_started":1}
I20260925 10:27:38.266835 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea333e7775441568814ae4a91a3e764" candidate_uuid: "65171ac073b441a1a76cacb24cccf937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811dfe326e8f48ae90b3ff9c796dc567" is_pre_election: true
I20260925 10:27:38.267525 25095 raft_consensus.cc:2454] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65171ac073b441a1a76cacb24cccf937 in term 0.
I20260925 10:27:38.267810 24818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea333e7775441568814ae4a91a3e764" candidate_uuid: "65171ac073b441a1a76cacb24cccf937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978236a5244e4fb2b860ac852a94076d" is_pre_election: true
I20260925 10:27:38.268285 24896 leader_election.cc:302] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65171ac073b441a1a76cacb24cccf937, 811dfe326e8f48ae90b3ff9c796dc567; no voters:
I20260925 10:27:38.268409 24818 raft_consensus.cc:2454] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65171ac073b441a1a76cacb24cccf937 in term 0.
I20260925 10:27:38.268797 25166 raft_consensus.cc:2781] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:27:38.268970 25166 raft_consensus.cc:482] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:27:38.269081 25166 raft_consensus.cc:3037] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:27:38.271453 25166 raft_consensus.cc:504] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.272055 25166 leader_election.cc:288] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [CANDIDATE]: Term 1 election: Requested vote from peers 811dfe326e8f48ae90b3ff9c796dc567 (127.12.248.67:35561), 978236a5244e4fb2b860ac852a94076d (127.12.248.65:40649)
I20260925 10:27:38.272615 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea333e7775441568814ae4a91a3e764" candidate_uuid: "65171ac073b441a1a76cacb24cccf937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811dfe326e8f48ae90b3ff9c796dc567"
I20260925 10:27:38.272711 24818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cea333e7775441568814ae4a91a3e764" candidate_uuid: "65171ac073b441a1a76cacb24cccf937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978236a5244e4fb2b860ac852a94076d"
I20260925 10:27:38.272838 25095 raft_consensus.cc:3037] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:27:38.272920 24818 raft_consensus.cc:3037] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:27:38.276544 24818 raft_consensus.cc:2454] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65171ac073b441a1a76cacb24cccf937 in term 1.
I20260925 10:27:38.276563 25095 raft_consensus.cc:2454] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65171ac073b441a1a76cacb24cccf937 in term 1.
I20260925 10:27:38.276963 24896 leader_election.cc:302] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65171ac073b441a1a76cacb24cccf937, 978236a5244e4fb2b860ac852a94076d; no voters:
I20260925 10:27:38.277263 25166 raft_consensus.cc:2781] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:27:38.277784 25166 raft_consensus.cc:686] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 1 LEADER]: Becoming Leader. State: Replica: 65171ac073b441a1a76cacb24cccf937, State: Running, Role: LEADER
I20260925 10:27:38.278278 25166 consensus_queue.cc:237] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } }
I20260925 10:27:38.283128 24649 catalog_manager.cc:5495] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65171ac073b441a1a76cacb24cccf937 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "65171ac073b441a1a76cacb24cccf937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65171ac073b441a1a76cacb24cccf937" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 40015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20260925 10:27:38.323864 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.318192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.12.248.66:40015)
0925 10:27:38.318470 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.318476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.318495 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:27:38.318709 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.318714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.318811 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.318973 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:27:38.319357 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.320162 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.321619 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.322072 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.322222 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.322729 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:27:38.323241 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:27:38.323428 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.323488 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.323570 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260925 10:27:38.328518 25141 tablet.cc:2369] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:27:38.333045 25095 raft_consensus.cc:1261] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 1 FOLLOWER]: Refusing update from remote peer 65171ac073b441a1a76cacb24cccf937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:27:38.333159 24818 raft_consensus.cc:1261] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 1 FOLLOWER]: Refusing update from remote peer 65171ac073b441a1a76cacb24cccf937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:27:38.334262 25170 consensus_queue.cc:1035] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "811dfe326e8f48ae90b3ff9c796dc567" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:27:38.334681 25166 consensus_queue.cc:1035] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978236a5244e4fb2b860ac852a94076d" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:27:38.361155 24650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:27:38.361438 24650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:27:38.365302 24650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35c169822b3c40a9891e3c4ff37eaee0: Sending DeleteTablet for 3 replicas of tablet cea333e7775441568814ae4a91a3e764
I20260925 10:27:38.366590 24941 tablet_service.cc:1514] Processing DeleteTablet for tablet cea333e7775441568814ae4a91a3e764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:27:38 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:52764
I20260925 10:27:38.366941 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet cea333e7775441568814ae4a91a3e764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:27:38 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:58474
I20260925 10:27:38.367641 24798 tablet_service.cc:1514] Processing DeleteTablet for tablet cea333e7775441568814ae4a91a3e764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:27:38 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:39314
I20260925 10:27:38.368141 25182 tablet_replica.cc:330] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: stopping tablet replica
I20260925 10:27:38.368420 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 24730
I20260925 10:27:38.368649 25182 raft_consensus.cc:2229] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:27:38.369524 25182 raft_consensus.cc:2258] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:27:38.374316 25184 tablet_replica.cc:330] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: stopping tablet replica
I20260925 10:27:38.374776 25184 raft_consensus.cc:2229] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:27:38.375322 25184 raft_consensus.cc:2258] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:27:38.376181 25183 tablet_replica.cc:330] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: stopping tablet replica
I20260925 10:27:38.376621 25183 raft_consensus.cc:2229] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:27:38.377068 25183 raft_consensus.cc:2258] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:27:38.377374 25182 ts_tablet_manager.cc:1905] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:27:38.378177 25184 ts_tablet_manager.cc:1905] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:27:38.378886 25183 ts_tablet_manager.cc:1905] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:27:38.385876 25184 ts_tablet_manager.cc:1918] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:27:38.386154 25184 log.cc:1196] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/cea333e7775441568814ae4a91a3e764
Sep 25 10:27:38 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:27:38.386754 25184 ts_tablet_manager.cc:1939] T cea333e7775441568814ae4a91a3e764 P 978236a5244e4fb2b860ac852a94076d: Deleting consensus metadata
I20260925 10:27:38.386937 25182 ts_tablet_manager.cc:1918] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:27:38.387101 25182 log.cc:1196] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/cea333e7775441568814ae4a91a3e764
I20260925 10:27:38.387601 25182 ts_tablet_manager.cc:1939] T cea333e7775441568814ae4a91a3e764 P 65171ac073b441a1a76cacb24cccf937: Deleting consensus metadata
I20260925 10:27:38.388872 25183 ts_tablet_manager.cc:1918] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:27:38.388883 24636 catalog_manager.cc:4841] TS 978236a5244e4fb2b860ac852a94076d (127.12.248.65:40649): tablet cea333e7775441568814ae4a91a3e764 (table test-table [id=3c16f34938884c7483bf4c91a0615b3f]) successfully deleted
I20260925 10:27:38.389042 25183 log.cc:1196] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/cea333e7775441568814ae4a91a3e764
I20260925 10:27:38.390020 25183 ts_tablet_manager.cc:1939] T cea333e7775441568814ae4a91a3e764 P 811dfe326e8f48ae90b3ff9c796dc567: Deleting consensus metadata
I20260925 10:27:38.390446 24635 catalog_manager.cc:4841] TS 65171ac073b441a1a76cacb24cccf937 (127.12.248.66:40015): tablet cea333e7775441568814ae4a91a3e764 (table test-table [id=3c16f34938884c7483bf4c91a0615b3f]) successfully deleted
I20260925 10:27:38.391757 24636 catalog_manager.cc:4841] TS 811dfe326e8f48ae90b3ff9c796dc567 (127.12.248.67:35561): tablet cea333e7775441568814ae4a91a3e764 (table test-table [id=3c16f34938884c7483bf4c91a0615b3f]) successfully deleted
I20260925 10:27:38.395635 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.370546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.12.248.65:40649)
0925 10:27:38.370735 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.370741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.373936 (+ 3195us) server_negotiation.cc:407] Connection header received
0925 10:27:38.374241 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.374246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.374338 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.374424 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:38.376370 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.377477 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.379340 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.379996 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.380095 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.387950 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:38.387966 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:38.387985 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:38.388027 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:38.392234 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.393272 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.393280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.393282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.393423 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.394019 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.394023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.394025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.394401 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:38.394625 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.395093 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.395292 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:27:38.396458 24858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:27:38.496340 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 24730
I20260925 10:27:38.520417 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 24869
Sep 25 10:27:38 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:27:38.537941 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.521218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.12.248.66:40015)
0925 10:27:38.521402 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.521409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.521461 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:27:38.521670 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.521677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.521765 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.521896 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:38.523272 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.524552 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.525891 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.526484 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.526619 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.530790 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:38.530811 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:38.530833 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:38.530916 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:38.535051 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.535713 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.535723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.535727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.535925 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.536266 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.536271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.536275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.536731 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:38.537065 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.537277 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.537522 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:27:38.538676 25001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:27:38.655635 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 24869
I20260925 10:27:38.673997 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 25012
Sep 25 10:27:38 dist-test-slave-f9rl krb5kdc[24174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:27:38.689616 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:27:38.674859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.12.248.67:35561)
0925 10:27:38.675045 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:27:38.675049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:27:38.675068 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:27:38.675218 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:27:38.675222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:27:38.675282 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:27:38.675376 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:27:38.676763 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.677590 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.678875 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:27:38.679253 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:27:38.679345 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:27:38.683812 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:27:38.683828 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:27:38.683844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:27:38.683877 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:27:38.686794 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.687478 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.687485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.687487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.687609 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:27:38.687942 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:27:38.687945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:27:38.687947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:27:38.688506 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:27:38.688732 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:27:38.688986 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:27:38.689151 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:27:38.690292 25135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:27:38.781422 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 25012
I20260925 10:27:38.798270 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 24618
I20260925 10:27:38.799454 24680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:27:38.916105 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 24618
I20260925 10:27:38.941859 13281 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:27:39.277571 13281 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:27:39.277783 13281 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:27:39.622862 13281 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:27:39.623 UTC [24204] LOG: received smart shutdown request
2026-09-25 10:27:39.627 UTC [24204] LOG: background worker "logical replication launcher" (PID 24213) exited with exit code 1
2026-09-25 10:27:39.627 UTC [24208] LOG: shutting down
2026-09-25 10:27:39.804 UTC [24204] LOG: database system is shut down
2026-09-25T10:27:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50630 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:27:40 dist-test-slave-f9rl krb5kdc[25198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:27:40 dist-test-slave-f9rl krb5kdc[25198](info): set up 2 sockets
Sep 25 10:27:40 dist-test-slave-f9rl krb5kdc[25198](info): commencing operation
krb5kdc: starting...
W20260925 10:27:41.047511 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:27:41 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:27:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangeradmin_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangeradmin_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangerlookup_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangerlookup_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
I20260925 10:27:41.380249 13281 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZKvIMz/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/postgres -l logfile start
2026-09-25 10:27:43.937 UTC [25228] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:27:43.937 UTC [25228] LOG: listening on IPv4 address "127.12.248.84", port 41241
2026-09-25 10:27:43.940 UTC [25228] LOG: listening on Unix socket "/tmp/.s.PGSQL.41241"
2026-09-25 10:27:43.962 UTC [25231] LOG: database system was shut down at 2026-09-25 10:27:42 UTC
2026-09-25 10:27:43.965 UTC [25228] LOG: database system is ready to accept connections
I20260925 10:27:44.954876 13281 mini_postgres.cc:96] Postgres bound to 41241
2026-09-25 10:27:44.972 UTC [25239] FATAL: database "slave" does not exist
127.12.248.84:41241 - accepting connections
I20260925 10:27:44.973084 13281 mini_ranger.cc:161] Starting Ranger...
I20260925 10:27:44.999641 13281 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:27:45.499979 13281 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:27:45.500098 13281 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin
2026-09-25 10:27:45,806 [I] DB FLAVOR :POSTGRES
2026-09-25 10:27:45,806 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:27:45,806 [I] Checking connection..
2026-09-25 10:27:45,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:27:46,069 [I] Checking connection passed.
2026-09-25 10:27:46,069 [I] --------- Verifying version history table ---------
2026-09-25 10:27:46,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:27:46,311 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:27:46,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:27:46,544 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:27:46,544 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:27:46,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:27:46.792 UTC [25303] WARNING: there is no transaction in progress
2026-09-25 10:27:46,799 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:27:46,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:27:47,077 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:27:47,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:27:47,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:27:47,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:27:47,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f9rl.c.gcp-upstream.internal','N') ;"
2026-09-25 10:27:47,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:27:48,088 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:27:48,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:27:48,347 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:27:48,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:27:48,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:27:48,857 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:27:48,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:27:49.810 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.825 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.842 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.857 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.872 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.889 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.943 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.950 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.964 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.975 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:49.986 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50.000 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50.010 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50.016 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50.023 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50.029 UTC [25411] WARNING: there is no transaction in progress
2026-09-25 10:27:50,699 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:27:50,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:27:50,948 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:27:50,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:27:51,207 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:27:51,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:27:51,480 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:27:51,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:27:51,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:27:51,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f9rl.c.gcp-upstream.internal';"
2026-09-25 10:27:52,239 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:27:52,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:27:52,508 [I] DB_PATCHES have already been applied
I20260925 10:27:52.514604 13281 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZKvIMz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:27:52.514694 13281 mini_ranger.cc:193] Using host: 127.12.248.84
I20260925 10:27:52.529533 13281 mini_ranger.cc:227] Ranger admin URL: http://127.12.248.84:52293
Sep 25, 2026 10:27:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:27:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:27:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:27:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:27:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:27:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:27:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:27:53.682170 13281 mini_ranger.cc:160] Time spent starting Ranger: real 8.709s user 0.007s sys 0.032s
Sep 25, 2026 10:28:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:28:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:28:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:28:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:28:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:28:15.892331 13281 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.248.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.248.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangerkms_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.248.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/rangerkms_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.248.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.248.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.248.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/HTTP_127.12.248.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:28:16 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332096, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:28:16.073901 13281 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:28:16.097944 13281 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:28:17.064764 13281 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:28:17.064891 13281 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms
I20260925 10:28:17.064950 13281 mini_ranger_kms.cc:227] Using postgres at 127.12.248.84:41241
2026-09-25 10:28:17,440 [I] DB FLAVOR :POSTGRES
2026-09-25 10:28:17,441 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:28:17,441 [I] Checking connection
2026-09-25 10:28:17,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:28:17,704 [I] connection success
2026-09-25 10:28:17,704 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:28:17,705 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:28:17,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:28:17,948 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:28:17,949 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:28:17,949 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:28:17,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.248.84:41241/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:28:18,248 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:28:18.779348 13281 mini_ranger_kms.cc:310] Created kms service
I20260925 10:28:18.960098 13281 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:28:19.005471 13281 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:28:19 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332096, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:19.405375 13281 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:28:19.405715 13281 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZKvIMz/build/asan/bin/postgresql.jar:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZKvIMz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:28:19.405771 13281 mini_ranger_kms.cc:240] Using host: 127.12.248.84
I20260925 10:28:19.427103 13281 mini_ranger_kms.cc:276] Ranger KMS PID: 25592
I20260925 10:28:19.427234 13281 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.248.84:46705
Sep 25, 2026 10:28:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:28:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:28:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:28:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:28:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:28:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:28:21.805155 13281 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.731s user 0.081s sys 0.312s
I20260925 10:28:21.805418 13281 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:28:21.805637 13281 mini_ranger_kms.cc:401] 127.12.248.84:46705/kms/v1/keys
Sep 25 10:28:22 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:28:22 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:28:22 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332101, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.248.84@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:28:25 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:28:25.250231 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37279
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37279
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:25.588872 25642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:25.589195 25642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:25.589288 25642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:25.596943 25642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:25.597069 25642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:25.597117 25642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:25.597168 25642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:25.597218 25642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:25.607723 25642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.248.84:46705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37279
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:25.609059 25642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:25.610478 25642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:25.622990 25647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:25.623019 25648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:25.623199 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:25.623992 25642 server_base.cc:1034] running on GCE node
I20260925 10:28:25.624744 25642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:25.626277 25642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:25.627491 25642 hybrid_clock.cc:648] HybridClock initialized: now 1790332105627441 us; error 33 us; skew 500 ppm
I20260925 10:28:25.627804 25642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:25 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:25.635640 25642 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:25.637315 25642 webserver.cc:460] Webserver started at http://127.12.248.126:34921/ using document root <none> and password file <none>
I20260925 10:28:25.638080 25642 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:25.638216 25642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:25.638566 25642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:28:25 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:25.689986 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "bc160d5d001a496fb05d1865c41e96bd"
format_stamp: "Formatted at 2026-09-25 10:28:25 on dist-test-slave-f9rl"
server_key: "046a7fcc06659bb6a298f4a476b4c6c5"
server_key_iv: "a2caae9a7e767475f7132d6e47291b7e"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:25.690857 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "bc160d5d001a496fb05d1865c41e96bd"
format_stamp: "Formatted at 2026-09-25 10:28:25 on dist-test-slave-f9rl"
server_key: "046a7fcc06659bb6a298f4a476b4c6c5"
server_key_iv: "a2caae9a7e767475f7132d6e47291b7e"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:25.696448 25642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:28:25.713331 25657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:25.715050 25642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:25.715265 25642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "bc160d5d001a496fb05d1865c41e96bd"
format_stamp: "Formatted at 2026-09-25 10:28:25 on dist-test-slave-f9rl"
server_key: "046a7fcc06659bb6a298f4a476b4c6c5"
server_key_iv: "a2caae9a7e767475f7132d6e47291b7e"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:25.715509 25642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:25.739519 25642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:25.743413 25642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:25.743752 25642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:25.762818 25642 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37279
I20260925 10:28:25.762842 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37279 every 8 connection(s)
I20260925 10:28:25.764294 25642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:25.765087 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 25642
I20260925 10:28:25.765321 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:28:25.766228 13281 external_mini_cluster.cc:1428] Setting key 2e4055e62c4fb19c88b2de8e5c9eecef
I20260925 10:28:25.768992 25709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:28:25 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:25.781839 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Bootstrap starting.
I20260925 10:28:25.786059 25709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:25.787361 25709 log.cc:824] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:25.790445 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: No bootstrap required, opened a new log
I20260925 10:28:25.790828 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:25.768295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.12.248.126:37279)
0925 10:28:25.769095 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:25.769111 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:25.769161 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:28:25.771506 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:25.771534 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:25.772042 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:25.772501 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:25.773757 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:25.775920 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:25.777239 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:25.778348 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:25.778536 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:25.782563 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:25.782574 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:25.782594 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:25.782638 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:25.786615 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:25.787315 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:25.787324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:25.787333 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:25.787535 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:25.787895 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:25.787903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:25.787909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:25.788536 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:25.788856 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:25.789330 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 10:28:25.789762 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":331,"threads_started":1}
I20260925 10:28:25.798650 25709 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:25.799039 25709 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:25.799152 25709 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc160d5d001a496fb05d1865c41e96bd, State: Initialized, Role: FOLLOWER
I20260925 10:28:25.799772 25709 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:25.800024 25709 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:25.800166 25709 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:25.800354 25709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:25.802564 25709 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:25.803069 25709 leader_election.cc:302] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc160d5d001a496fb05d1865c41e96bd; no voters:
I20260925 10:28:25.803750 25709 leader_election.cc:288] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:28:25.803859 25714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:25.805101 25714 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 LEADER]: Becoming Leader. State: Replica: bc160d5d001a496fb05d1865c41e96bd, State: Running, Role: LEADER
I20260925 10:28:25.805665 25714 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:25.806093 25709 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:25.808499 25715 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc160d5d001a496fb05d1865c41e96bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } } }
I20260925 10:28:25.808848 25715 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:25.809420 25716 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc160d5d001a496fb05d1865c41e96bd. Latest consensus state: current_term: 1 leader_uuid: "bc160d5d001a496fb05d1865c41e96bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } } }
I20260925 10:28:25.809818 25716 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:25.816129 25719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:25.816689 25709 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:28:25.830895 25719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:25.846822 25719 catalog_manager.cc:1347] Generated new cluster ID: 756f4b6081754be396e17d5894bb31b1
I20260925 10:28:25.846954 25719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:25.896680 25719 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:28:25.899006 25719 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:25.913904 25719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Generated new TSK 0
I20260925 10:28:25.915380 25719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:28:26.658105 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:28:26 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332106, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:28:27.108754 25754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:27.109282 25754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:27.109503 25754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:27.121608 25754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:27.121773 25754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:27.121977 25754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:28:27.148770 25754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:27.151070 25754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:27.153021 25754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:27.172948 25765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:27.173270 25767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:27.172931 25764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:27.175246 25754 server_base.cc:1034] running on GCE node
I20260925 10:28:27.176038 25754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:27.177166 25754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:27.178385 25754 hybrid_clock.cc:648] HybridClock initialized: now 1790332107178305 us; error 77 us; skew 500 ppm
I20260925 10:28:27.178737 25754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:27 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332107, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:27.192837 25754 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:28:27.195852 25754 webserver.cc:460] Webserver started at http://127.12.248.65:38515/ using document root <none> and password file <none>
I20260925 10:28:27.196934 25754 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:27.197094 25754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:27.197582 25754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:28:27 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332107, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:27.218235 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff"
format_stamp: "Formatted at 2026-09-25 10:28:27 on dist-test-slave-f9rl"
server_key: "b91f11e6e501f1d2a889e37de48f5cee"
server_key_iv: "01a601060f188cf90290b0868364a116"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:27.219492 25754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff"
format_stamp: "Formatted at 2026-09-25 10:28:27 on dist-test-slave-f9rl"
server_key: "b91f11e6e501f1d2a889e37de48f5cee"
server_key_iv: "01a601060f188cf90290b0868364a116"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:27.227196 25754 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 10:28:27.274549 25774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:27.277667 25754 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.007s sys 0.009s
I20260925 10:28:27.277998 25754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff"
format_stamp: "Formatted at 2026-09-25 10:28:27 on dist-test-slave-f9rl"
server_key: "b91f11e6e501f1d2a889e37de48f5cee"
server_key_iv: "01a601060f188cf90290b0868364a116"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:27.278301 25754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:27.345916 25754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:27.357671 25754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:27.358112 25754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:27.370854 25754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:27.379380 25754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:27.379585 25754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 10:28:27.379812 25754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:27.379953 25754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:28:27 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332106, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:28.087446 25887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:33821 every 8 connection(s)
I20260925 10:28:28.087580 25754 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:33821
I20260925 10:28:28.089648 25754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:28:28.097500 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 25754
I20260925 10:28:28.097785 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:28:28.099009 13281 external_mini_cluster.cc:1428] Setting key 93353bcccf2bdbf882a3c957cea576c4
Sep 25 10:28:28 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332106, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:28:28 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332107, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:28.145310 25891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:28.096864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:34223 (local address 127.12.248.126:37279)
0925 10:28:28.118373 (+ 21509us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:28.118384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:28.119991 (+ 1607us) server_negotiation.cc:407] Connection header received
0925 10:28:28.123503 (+ 3512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:28.123509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:28.123606 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:28.123697 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:28.127257 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.128448 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.131035 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.131714 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.131820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:28.137038 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:28.137046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:28.137051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:28.137107 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:28.140954 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:28.142165 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:28.142175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:28.142181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:28.142367 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:28.143012 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:28.143017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:28.143021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:28.143388 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:28.143542 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:28.144663 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0925 10:28:28.144900 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21297,"thread_start_us":23159,"threads_started":1}
I20260925 10:28:28.147974 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:28.093568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.65:34223)
0925 10:28:28.119245 (+ 25677us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:28.119325 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:28.123090 (+ 3765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:28.124342 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:28.124356 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:28.125074 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:28.126936 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:28.126964 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.128792 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.128797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:28.130832 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:28.130858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.132224 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.132231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:28.132550 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:28.133402 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:28.133467 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:28.136707 (+ 3240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:28.141191 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:28.141205 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:28.141230 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:28.141998 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:28.142558 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:28.142566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:28.142572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:28.142867 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:28.143657 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:28.143665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:28.144189 (+ 524us) client_negotiation.cc:769] Sending connection context
0925 10:28:28.145852 (+ 1663us) client_negotiation.cc:240] Negotiation successful
0925 10:28:28.146437 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25189,"thread_start_us":25230,"threads_started":1}
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:28:28.150420 25888 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:28.150780 25888 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:28.151698 25888 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
I20260925 10:28:28.154544 25674 ts_manager.cc:194] Registered new tserver with Master: 10e3212df2be4e9a8e4a3c4a3bd7eeff (127.12.248.65:33821)
I20260925 10:28:28.156575 25674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:34223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:28:28.189913 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:28.595038 25895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:28.595327 25895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:28.595481 25895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:28.603127 25895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:28.603258 25895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:28.603431 25895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:28:28.614207 25895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:28.615609 25895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:28.617084 25895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:28.632457 25908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:28.632499 25910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:28.632563 25907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:28.634145 25895 server_base.cc:1034] running on GCE node
I20260925 10:28:28.634826 25895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:28.635867 25895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:28.637132 25895 hybrid_clock.cc:648] HybridClock initialized: now 1790332108637048 us; error 48 us; skew 500 ppm
I20260925 10:28:28.637562 25895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:28 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332108, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:28.640775 25895 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:28:28.642547 25895 webserver.cc:460] Webserver started at http://127.12.248.66:43515/ using document root <none> and password file <none>
I20260925 10:28:28.643327 25895 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:28.643458 25895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:28.643790 25895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:28:28 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332108, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:28.662806 25895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "97a0632700754c4e82d58701659fecd4"
format_stamp: "Formatted at 2026-09-25 10:28:28 on dist-test-slave-f9rl"
server_key: "1a129a058dd5882c9943f47a5cec3bca"
server_key_iv: "8491d7bf9887292bf56e29021f74ad59"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:28.663995 25895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "97a0632700754c4e82d58701659fecd4"
format_stamp: "Formatted at 2026-09-25 10:28:28 on dist-test-slave-f9rl"
server_key: "1a129a058dd5882c9943f47a5cec3bca"
server_key_iv: "8491d7bf9887292bf56e29021f74ad59"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:28.671501 25895 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260925 10:28:28.690580 25917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:28.692714 25895 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:28:28.692951 25895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "97a0632700754c4e82d58701659fecd4"
format_stamp: "Formatted at 2026-09-25 10:28:28 on dist-test-slave-f9rl"
server_key: "1a129a058dd5882c9943f47a5cec3bca"
server_key_iv: "8491d7bf9887292bf56e29021f74ad59"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:28.693223 25895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:28.714777 25895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:28.719822 25895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:28.720183 25895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:28.721465 25895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:28.723237 25895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:28.723340 25895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:28.723465 25895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:28.723531 25895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:28.758247 25895 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:46671
I20260925 10:28:28.758266 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:46671 every 8 connection(s)
I20260925 10:28:28.759887 25895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:28:28.769949 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 25895
I20260925 10:28:28.770172 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:28:28.770918 13281 external_mini_cluster.cc:1428] Setting key 3038b02fa7ffa206b369de5076c611e0
Sep 25 10:28:28 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332108, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:28:28.801712 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:28.762600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:36397 (local address 127.12.248.126:37279)
0925 10:28:28.785824 (+ 23224us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:28.785829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:28.785850 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:28:28.786177 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:28.786186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:28.786279 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:28.786405 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:28.787872 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.789158 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.790858 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.791630 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.791775 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:28.795384 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:28.795389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:28.795392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:28.795426 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:28.798333 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:28.799205 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:28.799213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:28.799221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:28.799428 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:28.799903 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:28.799907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:28.799911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:28.800197 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:28.800358 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:28.801067 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 10:28:28.801300 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23115,"thread_start_us":22993,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:28:28.802821 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:28.763497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.66:36397)
0925 10:28:28.782320 (+ 18823us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:28.782391 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:28.785878 (+ 3487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:28.786626 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:28.786634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:28.787090 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:28.787688 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:28.787706 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.789322 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.789326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:28.790709 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:28.790721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:28.791843 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:28.791848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:28.792065 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:28.792766 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:28.792792 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:28.795149 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:28.798534 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:28.798544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:28.798562 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:28.799072 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:28.799537 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:28.799543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:28.799547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:28.799753 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:28.800361 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:28.800372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:28.800702 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 10:28:28.800979 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 10:28:28.801520 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18383,"thread_start_us":18075,"threads_started":1}
I20260925 10:28:28.804388 26030 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:28.804737 26030 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:28.805678 26030 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:28.807456 25674 ts_manager.cc:194] Registered new tserver with Master: 97a0632700754c4e82d58701659fecd4 (127.12.248.66:46671)
I20260925 10:28:28.808271 25674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:36397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:28:28.833859 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:29.160950 25888 heartbeater.cc:503] Master 127.12.248.126:37279 was elected leader, sending a full tablet report...
W20260925 10:28:29.172438 26036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:29.172730 26036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:29.172909 26036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:29.180299 26036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:29.180414 26036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:29.180601 26036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:28:29.190830 26036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:29.192250 26036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:29.193674 26036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:29.207243 26042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:29.207562 26044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:29.207228 26041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:29.208652 26036 server_base.cc:1034] running on GCE node
I20260925 10:28:29.209250 26036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:29.210355 26036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:29.211584 26036 hybrid_clock.cc:648] HybridClock initialized: now 1790332109211543 us; error 68 us; skew 500 ppm
I20260925 10:28:29.211912 26036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332109, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:29.215173 26036 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:28:29.216691 26036 webserver.cc:460] Webserver started at http://127.12.248.67:43701/ using document root <none> and password file <none>
I20260925 10:28:29.217414 26036 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:29.217628 26036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:29.217960 26036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332109, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:29.232569 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "bcff74fc11e147c089c4405975a126a3"
format_stamp: "Formatted at 2026-09-25 10:28:29 on dist-test-slave-f9rl"
server_key: "1ed5801eaede9398db220b2d14d81e9b"
server_key_iv: "6cc1f83181dde1796f50fcc0045c70b2"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:29.233570 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "bcff74fc11e147c089c4405975a126a3"
format_stamp: "Formatted at 2026-09-25 10:28:29 on dist-test-slave-f9rl"
server_key: "1ed5801eaede9398db220b2d14d81e9b"
server_key_iv: "6cc1f83181dde1796f50fcc0045c70b2"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:29.239403 26036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:28:29.255882 26051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:29.257586 26036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:29.257774 26036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "bcff74fc11e147c089c4405975a126a3"
format_stamp: "Formatted at 2026-09-25 10:28:29 on dist-test-slave-f9rl"
server_key: "1ed5801eaede9398db220b2d14d81e9b"
server_key_iv: "6cc1f83181dde1796f50fcc0045c70b2"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:29.258004 26036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:29.273005 26036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:29.276782 26036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:29.277133 26036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:29.278182 26036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:29.279561 26036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:29.279632 26036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:29.279806 26036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:29.279861 26036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:29.313084 26036 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:43399
I20260925 10:28:29.313120 26163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:43399 every 8 connection(s)
I20260925 10:28:29.314453 26036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:28:29.318385 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26036
I20260925 10:28:29.318576 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:28:29.319217 13281 external_mini_cluster.cc:1428] Setting key 34ffaa3484f4b9b2f10821073ef234b1
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332109, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:29.335131 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:29.317172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:60983 (local address 127.12.248.126:37279)
0925 10:28:29.317704 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:29.317709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:29.318845 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 10:28:29.320950 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:29.320954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:29.321049 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:29.321169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:29.322846 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.323617 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.325141 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.325612 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.325704 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:29.328835 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:29.328840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:29.328843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:29.328894 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:29.331595 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.332269 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.332273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.332275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.332405 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.332959 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.332964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.332967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.333417 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:29.333587 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:29.334473 (+ 886us) server_negotiation.cc:299] Negotiation successful
0925 10:28:29.334729 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260925 10:28:29.336054 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:29.317688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.67:60983)
0925 10:28:29.318507 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:29.318579 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:29.320623 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:29.321513 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:29.321526 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:29.321944 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:29.322696 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:29.322714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.323752 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.323756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:29.325014 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:29.325032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.325785 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.325789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:29.325981 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:29.326618 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:29.326639 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:29.328632 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:29.331723 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:29.331729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:29.331742 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:29.332163 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:29.332556 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:29.332560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:29.332563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:29.332822 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:29.333551 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:29.333557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:29.334016 (+ 459us) client_negotiation.cc:769] Sending connection context
0925 10:28:29.334361 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 10:28:29.334864 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260925 10:28:29.337529 26164 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:29.337826 26164 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:29.338562 26164 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:29.340065 25674 ts_manager.cc:194] Registered new tserver with Master: bcff74fc11e147c089c4405975a126a3 (127.12.248.67:43399)
I20260925 10:28:29.340884 25674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:60983
I20260925 10:28:29.347323 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:28:29.348970 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 25754
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:28:29.367180 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:29.349959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.12.248.65:33821)
0925 10:28:29.350605 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:29.350612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:29.350638 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:28:29.350691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:29.350697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:29.350902 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:29.351025 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:29.352451 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.353651 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.355164 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.355753 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.355851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:29.360344 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:29.360360 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:29.360376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:29.360424 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:29.363948 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.364772 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.364780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.364782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.364930 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.365353 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.365359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.365361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.365803 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:29.366158 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:29.366516 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:28:29.366804 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":310,"threads_started":1}
I20260925 10:28:29.368472 25882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:29.452201 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 25754
I20260925 10:28:29.468665 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 25895
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:28:29.490967 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:29.469917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.12.248.66:46671)
0925 10:28:29.470420 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:29.470428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:29.470455 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:28:29.470517 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:29.470523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:29.470748 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:29.470927 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:29.472329 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.473344 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.474844 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.475415 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.475563 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:29.484528 (+ 8965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:29.484545 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:29.484562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:29.484604 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:29.487667 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.488465 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.488473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.488476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.488615 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.489036 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.489039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.489041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.489514 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:29.489829 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:29.490175 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:28:29.490484 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260925 10:28:29.492359 26025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:29.577580 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 25895
I20260925 10:28:29.592962 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26036
Sep 25 10:28:29 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:28:29.610057 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:29.593988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.12.248.67:43399)
0925 10:28:29.594572 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:29.594579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:29.594607 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:28:29.594669 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:29.594675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:29.594882 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:29.595045 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:29.596256 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.597282 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.598645 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:29.599259 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:29.599385 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:29.603832 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:29.603845 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:29.603862 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:29.603900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:29.607045 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.607976 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.607984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.607986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.608123 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:29.608489 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:29.608493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:29.608495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:29.608844 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:29.609092 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:29.609351 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:28:29.609638 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
I20260925 10:28:29.611377 26159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:29.695847 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26036
I20260925 10:28:29.711548 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 25642
I20260925 10:28:29.712862 25704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:29.820070 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 25642
I20260925 10:28:29.841387 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37279
--webserver_interface=127.12.248.126
--webserver_port=34921
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37279
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:30.182629 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:30.182952 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:30.183121 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:30.190521 26176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:30.190656 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:30.190800 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:30.190897 26176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:30.190992 26176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:30.201462 26176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37279
--ranger_config_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=34921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:30.202883 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:30.204317 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:30.217276 26184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:30.217407 26181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:30.217381 26182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:30.218603 26176 server_base.cc:1034] running on GCE node
I20260925 10:28:30.219437 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:30.221107 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:30.222368 26176 hybrid_clock.cc:648] HybridClock initialized: now 1790332110222300 us; error 61 us; skew 500 ppm
I20260925 10:28:30.222718 26176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:30 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:30.226104 26176 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:30.227759 26176 webserver.cc:460] Webserver started at http://127.12.248.126:34921/ using document root <none> and password file <none>
I20260925 10:28:30.228411 26176 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:30.228530 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:28:30 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:30.297726 26176 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:28:30.300953 26192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:30.302632 26176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:30.302839 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "bc160d5d001a496fb05d1865c41e96bd"
format_stamp: "Formatted at 2026-09-25 10:28:25 on dist-test-slave-f9rl"
server_key: "046a7fcc06659bb6a298f4a476b4c6c5"
server_key_iv: "a2caae9a7e767475f7132d6e47291b7e"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:30.303865 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:30.322644 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:30.326807 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:30.327165 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:30.346684 26176 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37279
I20260925 10:28:30.346704 26243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37279 every 8 connection(s)
I20260925 10:28:30.348344 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:30.355705 26244 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:28:30.356536 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26176
I20260925 10:28:30.357416 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:33821
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:37279
--webserver_port=38515
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:30.360261 26244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Bootstrap starting.
I20260925 10:28:30.381687 26244 log.cc:824] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:30.391582 26244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:28:30.392179 26244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Bootstrap complete.
I20260925 10:28:30.402347 26244 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:30.403321 26244 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc160d5d001a496fb05d1865c41e96bd, State: Initialized, Role: FOLLOWER
I20260925 10:28:30.403970 26244 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:30.404245 26244 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:30.404400 26244 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:30.404604 26244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:28:30.409235 26244 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:30.409844 26244 leader_election.cc:302] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc160d5d001a496fb05d1865c41e96bd; no voters:
I20260925 10:28:30.410631 26244 leader_election.cc:288] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:28:30.410828 26248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:28:30.413259 26244 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:30.413739 26248 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [term 2 LEADER]: Becoming Leader. State: Replica: bc160d5d001a496fb05d1865c41e96bd, State: Running, Role: LEADER
I20260925 10:28:30.414413 26248 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } }
I20260925 10:28:30.419215 26250 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc160d5d001a496fb05d1865c41e96bd. Latest consensus state: current_term: 2 leader_uuid: "bc160d5d001a496fb05d1865c41e96bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } } }
I20260925 10:28:30.419731 26250 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:30.420418 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bc160d5d001a496fb05d1865c41e96bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc160d5d001a496fb05d1865c41e96bd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37279 } } }
I20260925 10:28:30.420759 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:30.422570 26257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:30.430275 26257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:30.431182 26244 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:28:30.432013 26257 catalog_manager.cc:1259] Loaded cluster ID: 756f4b6081754be396e17d5894bb31b1
I20260925 10:28:30.432194 26257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:30.436786 26257 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:30.438805 26257 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Loaded TSK: 0
I20260925 10:28:30.440622 26257 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:28:30.780920 26246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:30.781461 26246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:30.781701 26246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:30.793773 26246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:30.794014 26246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:30.794257 26246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:28:30.809056 26246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=38515
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:30.810837 26246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:30.812608 26246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:30.854059 26282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:30.855144 26283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:30.859104 26285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:30.860924 26246 server_base.cc:1034] running on GCE node
I20260925 10:28:30.861909 26246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:30.864240 26246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:30.867678 26246 hybrid_clock.cc:648] HybridClock initialized: now 1790332110867601 us; error 63 us; skew 500 ppm
I20260925 10:28:30.868211 26246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:30 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:30.879808 26246 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:28:30.889339 26246 webserver.cc:460] Webserver started at http://127.12.248.65:38515/ using document root <none> and password file <none>
I20260925 10:28:30.890566 26246 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:30.890765 26246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:28:30 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:30.932590 26246 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.006s
I20260925 10:28:30.954152 26293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:30.956565 26246 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.004s sys 0.016s
I20260925 10:28:30.956884 26246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff"
format_stamp: "Formatted at 2026-09-25 10:28:27 on dist-test-slave-f9rl"
server_key: "b91f11e6e501f1d2a889e37de48f5cee"
server_key_iv: "01a601060f188cf90290b0868364a116"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:30.958817 26246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:30.994871 26246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:31.000869 26246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:31.001339 26246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:31.013135 26246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:31.021852 26246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:31.022091 26246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:31.022324 26246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:31.022476 26246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:31.170872 26246 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:33821
I20260925 10:28:31.171005 26406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:33821 every 8 connection(s)
I20260925 10:28:31.172974 26246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:28:31.178282 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26246
I20260925 10:28:31.179494 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:46671
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:37279
--webserver_port=43515
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:28:31 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:31.230746 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:31.181751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.65:49749)
0925 10:28:31.193819 (+ 12068us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:31.193895 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:31.197822 (+ 3927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:31.200205 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:31.200218 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:31.200835 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:31.201410 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:31.201484 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.203408 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.203414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:31.204731 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:31.204775 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.206843 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.206851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:31.207184 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:31.209548 (+ 2364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:31.209585 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:31.212412 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:31.224990 (+ 12578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:31.225002 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:31.225020 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:31.225821 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:31.226496 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:31.226501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:31.226505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:31.226764 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:31.227904 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:31.227914 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:31.228399 (+ 485us) client_negotiation.cc:769] Sending connection context
0925 10:28:31.228704 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 10:28:31.229311 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11608,"thread_start_us":11278,"threads_started":1}
I20260925 10:28:31.233145 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:31.182919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49749 (local address 127.12.248.126:37279)
0925 10:28:31.192888 (+ 9969us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:31.192919 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:31.197568 (+ 4649us) server_negotiation.cc:407] Connection header received
0925 10:28:31.198956 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:31.198982 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:31.199583 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:31.200055 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:31.201752 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.203220 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.205859 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.206575 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.206810 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:31.212807 (+ 5997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:31.212825 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:31.212863 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:31.212933 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:31.224716 (+ 11783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:31.226056 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:31.226068 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:31.226076 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:31.226322 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:31.226910 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:31.226915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:31.226919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:31.227589 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:31.227788 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:31.231457 (+ 3669us) server_negotiation.cc:299] Negotiation successful
0925 10:28:31.231918 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9577,"thread_start_us":9273,"threads_started":1}
I20260925 10:28:31.237682 26407 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:31.238073 26407 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:31.239064 26407 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
W20260925 10:28:31.242669 26407 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:37279 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:28:31.242935 26407 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:31.243273 26407 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:31.244596 26407 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:31.244896 26407 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
W20260925 10:28:31.246083 26407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:28:31.624284 26411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:31.624752 26411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:31.624939 26411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:31.636622 26411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:31.636826 26411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:31.637048 26411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:28:31.652853 26411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:46671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=43515
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:31.654625 26411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:31.656332 26411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:31.689411 26428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:31.689469 26427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:31.693605 26411 server_base.cc:1034] running on GCE node
W20260925 10:28:31.694026 26430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:31.694799 26411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:31.696079 26411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:31.697371 26411 hybrid_clock.cc:648] HybridClock initialized: now 1790332111697286 us; error 72 us; skew 500 ppm
I20260925 10:28:31.697777 26411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:31 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:31.701964 26411 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:28:31.707006 26411 webserver.cc:460] Webserver started at http://127.12.248.66:43515/ using document root <none> and password file <none>
I20260925 10:28:31.707947 26411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:31.708146 26411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:28:31 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
Sep 25 10:28:31 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:31.758442 26411 fs_manager.cc:709] Time spent opening directory manager: real 0.028s user 0.007s sys 0.004s
I20260925 10:28:31.771680 26438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:31.775972 26411 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.006s sys 0.004s
I20260925 10:28:31.776232 26411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "97a0632700754c4e82d58701659fecd4"
format_stamp: "Formatted at 2026-09-25 10:28:28 on dist-test-slave-f9rl"
server_key: "1a129a058dd5882c9943f47a5cec3bca"
server_key_iv: "8491d7bf9887292bf56e29021f74ad59"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:31.777787 26411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:31.813252 26411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:31.818830 26411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:31.819160 26411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:31.820436 26411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:31.822223 26411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:31.822340 26411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:31.822446 26411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:31.822520 26411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:31.897996 26411 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:46671
I20260925 10:28:31.898087 26551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:46671 every 8 connection(s)
I20260925 10:28:31.900092 26411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:28:31.905788 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26411
I20260925 10:28:31.906574 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.248.84:46705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:43399
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:37279
--webserver_port=43701
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:28:31 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:31.941811 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:31.904883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.66:41807)
0925 10:28:31.906443 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:31.906495 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:31.909805 (+ 3310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:31.914104 (+ 4299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:31.914119 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:31.914821 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:31.915683 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:31.915712 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.917306 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.917314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:31.919623 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:31.919643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.921026 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.921035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:31.921375 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:31.922378 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:31.922419 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:31.930313 (+ 7894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:31.935735 (+ 5422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:31.935750 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:31.935774 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:31.936533 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:31.937052 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:31.937060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:31.937067 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:31.937384 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:31.938307 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:31.938319 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:31.939350 (+ 1031us) client_negotiation.cc:769] Sending connection context
0925 10:28:31.939718 (+ 368us) client_negotiation.cc:240] Negotiation successful
0925 10:28:31.940345 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"thread_start_us":8044,"threads_started":1}
I20260925 10:28:31.942935 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:31.912865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:41807 (local address 127.12.248.126:37279)
0925 10:28:31.913345 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:31.913351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:31.913372 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:28:31.913472 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:31.913478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:31.913566 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:31.913679 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:31.915920 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.917117 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.919819 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:31.920546 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:31.920682 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:31.930659 (+ 9977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:31.930668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:31.930673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:31.930726 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:31.935509 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:31.936720 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:31.936729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:31.936734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:31.936898 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:31.937603 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:31.937612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:31.937618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:31.938050 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:31.938223 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:31.942311 (+ 4088us) server_negotiation.cc:299] Negotiation successful
0925 10:28:31.942570 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":945,"threads_started":1}
I20260925 10:28:31.945013 26552 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:31.945365 26552 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:31.946178 26552 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:31.948191 26209 ts_manager.cc:194] Registered new tserver with Master: 97a0632700754c4e82d58701659fecd4 (127.12.248.66:46671)
I20260925 10:28:31.950474 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:41807
Sep 25 10:28:32 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:32.246240 26407 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:32.246699 26407 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:32.248035 26209 ts_manager.cc:194] Registered new tserver with Master: 10e3212df2be4e9a8e4a3c4a3bd7eeff (127.12.248.65:33821)
I20260925 10:28:32.249346 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:49749
W20260925 10:28:32.368136 26556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:32.368714 26556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:32.368924 26556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:32.379987 26556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:32.380219 26556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:32.380460 26556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:28:32.398847 26556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.248.84:46705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:43399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=43701
--tserver_master_addrs=127.12.248.126:37279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:32.400811 26556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:32.402631 26556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:32.439379 26562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:32.441386 26561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:32.444057 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:32.444660 26556 server_base.cc:1034] running on GCE node
I20260925 10:28:32.445732 26556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:32.461915 26556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:32.463282 26556 hybrid_clock.cc:648] HybridClock initialized: now 1790332112463202 us; error 83 us; skew 500 ppm
I20260925 10:28:32.463861 26556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:32 dist-test-slave-f9rl krb5kdc[25198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:32.468374 26556 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:28:32.496891 26556 webserver.cc:460] Webserver started at http://127.12.248.67:43701/ using document root <none> and password file <none>
I20260925 10:28:32.498059 26556 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:32.498342 26556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:28:32 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:32.608506 26556 fs_manager.cc:709] Time spent opening directory manager: real 0.049s user 0.003s sys 0.012s
Sep 25 10:28:32 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332111, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for HTTP/127.12.248.84@KRBTEST.COM
I20260925 10:28:32.626310 26573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:32.629602 26556 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.003s sys 0.002s
I20260925 10:28:32.629858 26556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "bcff74fc11e147c089c4405975a126a3"
format_stamp: "Formatted at 2026-09-25 10:28:29 on dist-test-slave-f9rl"
server_key: "1ed5801eaede9398db220b2d14d81e9b"
server_key_iv: "6cc1f83181dde1796f50fcc0045c70b2"
server_key_version: "kuduclusterkey@0"
I20260925 10:28:32.631553 26556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:32.713658 26556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:32.719411 26556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:32.719956 26556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:32.745976 26556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:32.754611 26556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:32.754837 26556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 10:28:32.755041 26556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:32.755172 26556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:32.907235 26556 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:43399
I20260925 10:28:32.907279 26690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:43399 every 8 connection(s)
I20260925 10:28:32.909060 26556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:28:32.917577 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26556
Sep 25 10:28:32 dist-test-slave-f9rl krb5kdc[25198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:32.939062 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:32.915428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:35153 (local address 127.12.248.126:37279)
0925 10:28:32.915933 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:32.915939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:32.917075 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 10:28:32.920124 (+ 3049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:32.920134 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:32.920272 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:32.920385 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:32.923240 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.924797 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.926886 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.927533 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.927626 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:32.931684 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:32.931689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:32.931693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:32.931735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:32.935501 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.936293 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.936300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.936309 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.936443 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.937061 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.937068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.937071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.937470 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:32.937660 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:32.938406 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 10:28:32.938637 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260925 10:28:32.940116 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:32.916091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37279 (local address 127.12.248.67:35153)
0925 10:28:32.916886 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:32.916939 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:32.919704 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:32.921107 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:32.921124 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:32.921967 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:32.922977 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:32.923006 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.925069 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.925077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:32.926742 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:32.926760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.927790 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.927798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:32.928117 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:32.928856 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:32.928893 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:32.931406 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:32.935666 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:32.935674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:32.935690 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:32.936132 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:32.936589 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:32.936597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:32.936600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:32.936913 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:32.937660 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:32.937668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:32.938020 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 10:28:32.938298 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 10:28:32.938987 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":338,"threads_started":1}
I20260925 10:28:32.941882 26691 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37279
I20260925 10:28:32.941932 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:32.918550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.12.248.126:37279)
0925 10:28:32.926807 (+ 8257us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:32.926836 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:32.926856 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:28:32.926897 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:32.926901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:32.927017 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:32.927122 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:32.929230 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.930177 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.932065 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.932626 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.932764 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:32.936170 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:32.936176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:32.936179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:32.936211 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:32.939079 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.939917 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.939925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.939929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.940065 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.940451 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.940457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.940460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.940778 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:32.941058 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:32.941415 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 10:28:32.941637 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8153,"thread_start_us":8008,"threads_started":1}
I20260925 10:28:32.942238 26691 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:32.943194 26691 heartbeater.cc:511] Master 127.12.248.126:37279 requested a full tablet report, sending...
I20260925 10:28:32.944967 26209 ts_manager.cc:194] Registered new tserver with Master: bcff74fc11e147c089c4405975a126a3 (127.12.248.67:43399)
I20260925 10:28:32.945948 26209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:35153
I20260925 10:28:32.955049 26552 heartbeater.cc:503] Master 127.12.248.126:37279 was elected leader, sending a full tablet report...
I20260925 10:28:32.957429 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:28:32.975634 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:32.961274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.12.248.126:37279)
0925 10:28:32.961630 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:32.961637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:32.961885 (+ 248us) server_negotiation.cc:407] Connection header received
0925 10:28:32.962103 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:32.962108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:32.962200 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:32.962325 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:32.964003 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.964801 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.966574 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:32.967045 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:32.967203 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:32.969918 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:32.969932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:32.969937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:32.969973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:32.972542 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.973346 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.973352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.973357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.973565 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:32.974122 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:32.974127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:32.974131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:32.974425 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:32.974691 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:32.975056 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 10:28:32.975295 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260925 10:28:32.980574 26209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:28:32.988832 26209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:28:33.016418 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.002995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:41668)
0925 10:28:33.003839 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.003886 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:33.004090 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:33.005587 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:33.005598 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:33.005641 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:33.006244 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.006261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.009204 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.009213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.012475 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.012498 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.015175 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.015185 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.015499 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.015581 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 10:28:33.015717 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:28:33.015993 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":396,"threads_started":1}
I20260925 10:28:33.017854 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.003294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.12.248.65:33821)
0925 10:28:33.004752 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.004764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.004799 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:28:33.004924 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.004935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.005249 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.006496 (+ 1247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:33.006779 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.008994 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.012765 (+ 3771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.014849 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.016961 (+ 2112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.017256 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.017341 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.017533 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":1074,"threads_started":1}
I20260925 10:28:33.021829 26342 tablet_service.cc:1467] Processing CreateTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 (DEFAULT_TABLE table=test-table [id=c0f345ea5f434f9e83e003c72ce52930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:33.023445 26342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:33.024403 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.003748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:46671 (local address 127.0.0.1:35192)
0925 10:28:33.016684 (+ 12936us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.016703 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:33.016803 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:33.018315 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:33.018320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:33.018346 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:33.018548 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.018558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.020433 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.020438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.022017 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.022029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.023414 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.023419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.023730 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.023756 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:28:33.023865 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:28:33.024004 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12846,"mutex_wait_us":1,"thread_start_us":13516,"threads_started":1}
I20260925 10:28:33.024403 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.004174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.12.248.66:46671)
0925 10:28:33.017551 (+ 13377us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.017560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.017592 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:28:33.017692 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.017700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.018084 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.018326 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:33.018741 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.020283 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.022177 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.023247 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.023401 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.023645 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.023911 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.024073 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12170,"thread_start_us":12013,"threads_started":1}
I20260925 10:28:33.026191 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.004198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:43399 (local address 127.0.0.1:60028)
0925 10:28:33.016920 (+ 12722us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.016940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:33.017073 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:33.018643 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:33.018647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:33.018673 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:33.019051 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.019078 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.021145 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.021150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.023290 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.023304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.025405 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.025412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.025612 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.025657 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:28:33.025724 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:28:33.025845 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12569,"thread_start_us":13128,"threads_started":1}
I20260925 10:28:33.027105 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.004942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.12.248.67:43399)
0925 10:28:33.017723 (+ 12781us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.017733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.017766 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:28:33.017881 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.017889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.018115 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.018325 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:33.019606 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.020983 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.023489 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.025150 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.026286 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.026512 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.026587 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.026748 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12576,"thread_start_us":12810,"threads_started":1}
I20260925 10:28:33.028533 26487 tablet_service.cc:1467] Processing CreateTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 (DEFAULT_TABLE table=test-table [id=c0f345ea5f434f9e83e003c72ce52930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:33.029736 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:33.030647 26621 tablet_service.cc:1467] Processing CreateTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 (DEFAULT_TABLE table=test-table [id=c0f345ea5f434f9e83e003c72ce52930]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:33.032130 26621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:33.043535 26714 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Bootstrap starting.
I20260925 10:28:33.045361 26715 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Bootstrap starting.
I20260925 10:28:33.047308 26714 tablet_bootstrap.cc:654] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:33.048559 26714 log.cc:824] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:33.048782 26715 tablet_bootstrap.cc:654] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:33.048771 26716 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Bootstrap starting.
I20260925 10:28:33.050254 26715 log.cc:824] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:33.052079 26714 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: No bootstrap required, opened a new log
I20260925 10:28:33.052148 26716 tablet_bootstrap.cc:654] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:33.052366 26714 ts_tablet_manager.cc:1397] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 10:28:33.052855 26715 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: No bootstrap required, opened a new log
I20260925 10:28:33.053100 26715 ts_tablet_manager.cc:1397] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:28:33.053355 26716 log.cc:824] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:33.056447 26716 tablet_bootstrap.cc:492] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: No bootstrap required, opened a new log
I20260925 10:28:33.056711 26716 ts_tablet_manager.cc:1397] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:28:33.060896 26715 raft_consensus.cc:348] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.061290 26715 raft_consensus.cc:374] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:33.061383 26715 raft_consensus.cc:729] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a0632700754c4e82d58701659fecd4, State: Initialized, Role: FOLLOWER
I20260925 10:28:33.062070 26715 consensus_queue.cc:260] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.063632 26714 raft_consensus.cc:348] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.064095 26714 raft_consensus.cc:374] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:33.064199 26714 raft_consensus.cc:729] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e3212df2be4e9a8e4a3c4a3bd7eeff, State: Initialized, Role: FOLLOWER
I20260925 10:28:33.064894 26715 ts_tablet_manager.cc:1428] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260925 10:28:33.065624 26714 consensus_queue.cc:260] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.066918 26407 heartbeater.cc:503] Master 127.12.248.126:37279 was elected leader, sending a full tablet report...
I20260925 10:28:33.067605 26714 ts_tablet_manager.cc:1428] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260925 10:28:33.067339 26716 raft_consensus.cc:348] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.067776 26716 raft_consensus.cc:374] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:33.067875 26716 raft_consensus.cc:729] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcff74fc11e147c089c4405975a126a3, State: Initialized, Role: FOLLOWER
I20260925 10:28:33.068475 26716 consensus_queue.cc:260] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.069649 26691 heartbeater.cc:503] Master 127.12.248.126:37279 was elected leader, sending a full tablet report...
I20260925 10:28:33.070436 26716 ts_tablet_manager.cc:1428] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 10:28:33.100654 26720 raft_consensus.cc:482] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:33.100919 26720 raft_consensus.cc:504] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.102581 26720 leader_election.cc:288] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10e3212df2be4e9a8e4a3c4a3bd7eeff (127.12.248.65:33821), bcff74fc11e147c089c4405975a126a3 (127.12.248.67:43399)
I20260925 10:28:33.110287 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.102996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:46389 (local address 127.12.248.65:33821)
0925 10:28:33.103228 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.103234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.103816 (+ 582us) server_negotiation.cc:407] Connection header received
0925 10:28:33.104437 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.104443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.104520 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.104572 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:33.105312 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.106225 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.108229 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.109239 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.109372 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.109424 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.109872 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.110034 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:28:33.110389 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.104077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:54873 (local address 127.12.248.67:43399)
0925 10:28:33.104311 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.104318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.104648 (+ 330us) server_negotiation.cc:407] Connection header received
0925 10:28:33.104842 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.104848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.104916 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.105023 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:33.105691 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.106519 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.108458 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.109395 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.109623 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.109696 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.110016 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.110155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:28:33.110494 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.103840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:43399 (local address 127.12.248.66:54873)
0925 10:28:33.104562 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.104579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:33.104705 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:33.105041 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:33.105045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:33.105068 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:33.105541 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.105553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.106654 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.106661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.108285 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.108299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.109663 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.109668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.109849 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.109883 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:28:33.109973 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:28:33.110091 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":39,"thread_start_us":416,"threads_started":1}
I20260925 10:28:33.110855 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.102986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.12.248.66:46389)
0925 10:28:33.103711 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.103732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:33.103949 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:33.104818 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:33.104822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:33.104843 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:33.105162 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.105170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.106357 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.106363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.108075 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:33.108087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.109469 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.109479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:33.109665 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.109733 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 10:28:33.109817 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:28:33.109912 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":6,"thread_start_us":406,"threads_started":1}
I20260925 10:28:33.111579 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c3c830bd2f48c8bf83a4a0d67d23a0" candidate_uuid: "97a0632700754c4e82d58701659fecd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcff74fc11e147c089c4405975a126a3" is_pre_election: true
I20260925 10:28:33.111745 26362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c3c830bd2f48c8bf83a4a0d67d23a0" candidate_uuid: "97a0632700754c4e82d58701659fecd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" is_pre_election: true
I20260925 10:28:33.112133 26641 raft_consensus.cc:2454] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97a0632700754c4e82d58701659fecd4 in term 0.
I20260925 10:28:33.112310 26362 raft_consensus.cc:2454] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97a0632700754c4e82d58701659fecd4 in term 0.
I20260925 10:28:33.112823 26442 leader_election.cc:302] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97a0632700754c4e82d58701659fecd4, bcff74fc11e147c089c4405975a126a3; no voters:
I20260925 10:28:33.113250 26720 raft_consensus.cc:2781] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:28:33.113407 26720 raft_consensus.cc:482] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:28:33.113541 26720 raft_consensus.cc:3037] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:33.115897 26720 raft_consensus.cc:504] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.116479 26720 leader_election.cc:288] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [CANDIDATE]: Term 1 election: Requested vote from peers 10e3212df2be4e9a8e4a3c4a3bd7eeff (127.12.248.65:33821), bcff74fc11e147c089c4405975a126a3 (127.12.248.67:43399)
I20260925 10:28:33.116979 26362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c3c830bd2f48c8bf83a4a0d67d23a0" candidate_uuid: "97a0632700754c4e82d58701659fecd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff"
I20260925 10:28:33.117005 26641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c3c830bd2f48c8bf83a4a0d67d23a0" candidate_uuid: "97a0632700754c4e82d58701659fecd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcff74fc11e147c089c4405975a126a3"
I20260925 10:28:33.117189 26362 raft_consensus.cc:3037] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:33.117215 26641 raft_consensus.cc:3037] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:33.119405 26362 raft_consensus.cc:2454] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a0632700754c4e82d58701659fecd4 in term 1.
I20260925 10:28:33.119464 26641 raft_consensus.cc:2454] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a0632700754c4e82d58701659fecd4 in term 1.
I20260925 10:28:33.119827 26442 leader_election.cc:302] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e3212df2be4e9a8e4a3c4a3bd7eeff, 97a0632700754c4e82d58701659fecd4; no voters:
I20260925 10:28:33.120157 26720 raft_consensus.cc:2781] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:33.120563 26720 raft_consensus.cc:686] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 1 LEADER]: Becoming Leader. State: Replica: 97a0632700754c4e82d58701659fecd4, State: Running, Role: LEADER
I20260925 10:28:33.121062 26720 consensus_queue.cc:237] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } }
I20260925 10:28:33.126135 26208 catalog_manager.cc:5495] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97a0632700754c4e82d58701659fecd4 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "97a0632700754c4e82d58701659fecd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a0632700754c4e82d58701659fecd4" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 46671 } health_report { overall_health: HEALTHY } } }
W20260925 10:28:33.163443 26553 tablet.cc:2369] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:28:33.164400 26694 tablet.cc:2369] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:28:33.171674 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.166338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.12.248.66:46671)
0925 10:28:33.166562 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.166567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.166585 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:28:33.166733 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.166736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.166824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.166947 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:28:33.167292 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.168040 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.169337 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.169824 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.169920 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.170555 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:28:33.171065 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:28:33.171174 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.171301 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.171366 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:28:33.180712 26641 raft_consensus.cc:1261] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 1 FOLLOWER]: Refusing update from remote peer 97a0632700754c4e82d58701659fecd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:33.180794 26362 raft_consensus.cc:1261] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 1 FOLLOWER]: Refusing update from remote peer 97a0632700754c4e82d58701659fecd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:33.181810 26725 consensus_queue.cc:1035] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcff74fc11e147c089c4405975a126a3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:33.182211 26720 consensus_queue.cc:1035] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e3212df2be4e9a8e4a3c4a3bd7eeff" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:33.209820 26209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:33.210105 26209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:33.213869 26209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc160d5d001a496fb05d1865c41e96bd: Sending DeleteTablet for 3 replicas of tablet 11c3c830bd2f48c8bf83a4a0d67d23a0
I20260925 10:28:33.215296 26621 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:60028
I20260925 10:28:33.215307 26342 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41668
I20260925 10:28:33.216594 26487 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:35192
I20260925 10:28:33.221279 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26246
I20260925 10:28:33.228251 26739 tablet_replica.cc:330] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: stopping tablet replica
I20260925 10:28:33.228863 26739 raft_consensus.cc:2229] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:33.229570 26739 raft_consensus.cc:2258] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:33.230432 26740 tablet_replica.cc:330] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: stopping tablet replica
I20260925 10:28:33.230957 26740 raft_consensus.cc:2229] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:28:33.231784 26740 raft_consensus.cc:2258] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:33.232822 26738 tablet_replica.cc:330] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: stopping tablet replica
I20260925 10:28:33.233186 26738 raft_consensus.cc:2229] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:33.233717 26738 raft_consensus.cc:2258] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:33.235666 26739 ts_tablet_manager.cc:1905] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:33.235867 26738 ts_tablet_manager.cc:1905] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:33.239507 26740 ts_tablet_manager.cc:1905] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:33.242991 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.222801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.12.248.65:33821)
0925 10:28:33.223032 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.223042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.223945 (+ 903us) server_negotiation.cc:407] Connection header received
0925 10:28:33.224193 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.224201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.224319 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.224456 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:33.225749 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.227285 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.228660 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.229716 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.229859 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.233762 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:33.233778 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:33.233798 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:33.233869 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:33.238339 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.239314 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.239323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.239327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.239512 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.240004 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.240010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.240015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.240557 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:33.240930 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.242315 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.242560 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 10:28:33.243157 26738 ts_tablet_manager.cc:1918] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:33.243362 26738 log.cc:1196] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/11c3c830bd2f48c8bf83a4a0d67d23a0
I20260925 10:28:33.243357 26739 ts_tablet_manager.cc:1918] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:33.243546 26739 log.cc:1196] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/11c3c830bd2f48c8bf83a4a0d67d23a0
I20260925 10:28:33.244017 26402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:33.244114 26738 ts_tablet_manager.cc:1939] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P bcff74fc11e147c089c4405975a126a3: Deleting consensus metadata
I20260925 10:28:33.246423 26196 catalog_manager.cc:4841] TS bcff74fc11e147c089c4405975a126a3 (127.12.248.67:43399): tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 (table test-table [id=c0f345ea5f434f9e83e003c72ce52930]) successfully deleted
I20260925 10:28:33.249491 26740 ts_tablet_manager.cc:1918] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:33.249666 26740 log.cc:1196] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/11c3c830bd2f48c8bf83a4a0d67d23a0
I20260925 10:28:33.250221 26740 ts_tablet_manager.cc:1939] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 97a0632700754c4e82d58701659fecd4: Deleting consensus metadata
I20260925 10:28:33.252110 26195 catalog_manager.cc:4841] TS 97a0632700754c4e82d58701659fecd4 (127.12.248.66:46671): tablet 11c3c830bd2f48c8bf83a4a0d67d23a0 (table test-table [id=c0f345ea5f434f9e83e003c72ce52930]) successfully deleted
I20260925 10:28:33.336369 26739 ts_tablet_manager.cc:1939] T 11c3c830bd2f48c8bf83a4a0d67d23a0 P 10e3212df2be4e9a8e4a3c4a3bd7eeff: Deleting consensus metadata
I20260925 10:28:33.337703 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26246
W20260925 10:28:33.360383 26196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.248.65:33821 (error 108)
I20260925 10:28:33.361297 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26411
I20260925 10:28:33.363906 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.363403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42368)
0925 10:28:33.363592 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.363723 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:28:33.364220 26196 catalog_manager.cc:4584] TS 10e3212df2be4e9a8e4a3c4a3bd7eeff (127.12.248.65:33821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11c3c830bd2f48c8bf83a4a0d67d23a0: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
I20260925 10:28:33.378414 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.362888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.12.248.66:46671)
0925 10:28:33.363114 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.363121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.363143 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:28:33.363210 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.363216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.363328 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.363449 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:33.365422 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.366584 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.367970 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.368505 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.368658 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.371557 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:33.371571 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:33.371587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:33.371627 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:33.375185 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.375971 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.375987 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.375990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.376164 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.376608 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.376614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.376618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.376991 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:33.377400 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.377721 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.377981 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:28:33.379220 26547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:33.418357 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.417779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42370)
0925 10:28:33.418053 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.418164 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 10:28:33.419503 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.419114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42372)
0925 10:28:33.419282 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.419366 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:28:33.480122 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.479591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42380)
0925 10:28:33.479862 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.479959 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 10:28:33.481252 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.480902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42394)
0925 10:28:33.481048 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.481131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:28:33.495043 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26411
I20260925 10:28:33.513511 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26556
I20260925 10:28:33.527076 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.514330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.12.248.67:43399)
0925 10:28:33.514552 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:33.514557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:33.514576 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:28:33.514661 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:33.514664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:33.514726 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:33.514808 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:33.516106 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.516886 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.518198 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:33.518590 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:33.518683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:33.521208 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:33.521219 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:33.521235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:33.521278 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:33.524197 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.524847 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.524854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.524856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.524999 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:33.525487 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:33.525497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:33.525506 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:33.525945 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:33.526204 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:33.526486 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:28:33.526664 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:28:33.527772 26684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:33.562093 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.561582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42400)
0925 10:28:33.561800 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.561902 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 10:28:33.563211 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.562821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42410)
0925 10:28:33.563031 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.563104 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 10:28:33.620064 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26556
I20260925 10:28:33.637199 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26176
I20260925 10:28:33.638479 26239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:33.754848 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:33.751779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:33821 (local address 127.0.0.1:42420)
0925 10:28:33.754237 (+ 2458us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:33.754385 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2278}
I20260925 10:28:33.755193 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26176
I20260925 10:28:33.780256 13281 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:28:34.115711 13281 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:28:34.115911 13281 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:28:34.461740 13281 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:28:34.462 UTC [25228] LOG: received smart shutdown request
2026-09-25 10:28:34.465 UTC [25228] LOG: background worker "logical replication launcher" (PID 25237) exited with exit code 1
2026-09-25 10:28:34.465 UTC [25232] LOG: shutting down
2026-09-25 10:28:34.652 UTC [25228] LOG: database system is shut down
2026-09-25T10:28:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54845 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:28:34 dist-test-slave-f9rl krb5kdc[26754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:28:34 dist-test-slave-f9rl krb5kdc[26754](info): set up 2 sockets
Sep 25 10:28:34 dist-test-slave-f9rl krb5kdc[26754](info): commencing operation
krb5kdc: starting...
W20260925 10:28:35.919354 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:28:36 dist-test-slave-f9rl krb5kdc[26754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:28:36.160876 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38161
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:36.504529 26767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:36.504920 26767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:36.505102 26767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:36.512569 26767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:36.512749 26767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:36.512883 26767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:36.512991 26767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:36.513090 26767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:36.523473 26767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:36.525002 26767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:36.526558 26767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:36.539485 26772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:36.539549 26775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:36.539510 26773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:36.540421 26767 server_base.cc:1034] running on GCE node
I20260925 10:28:36.541276 26767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:36.542831 26767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:36.544066 26767 hybrid_clock.cc:648] HybridClock initialized: now 1790332116544041 us; error 50 us; skew 500 ppm
I20260925 10:28:36.544410 26767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:36 dist-test-slave-f9rl krb5kdc[26754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:36.553313 26767 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:36.555354 26767 webserver.cc:460] Webserver started at http://127.12.248.126:41029/ using document root <none> and password file <none>
I20260925 10:28:36.556115 26767 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:36.556269 26767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:36.556602 26767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:36.559870 26767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "9c197defe3f74743827d1c6aae65bbf5"
format_stamp: "Formatted at 2026-09-25 10:28:36 on dist-test-slave-f9rl"
server_key: "b321d59db0df2d9753efd53a1accf0cc"
server_key_iv: "abd3a3242cccf9cef367c52c5d9ca6de"
server_key_version: "encryptionkey@0"
I20260925 10:28:36.560720 26767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "9c197defe3f74743827d1c6aae65bbf5"
format_stamp: "Formatted at 2026-09-25 10:28:36 on dist-test-slave-f9rl"
server_key: "b321d59db0df2d9753efd53a1accf0cc"
server_key_iv: "abd3a3242cccf9cef367c52c5d9ca6de"
server_key_version: "encryptionkey@0"
I20260925 10:28:36.566308 26767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 10:28:36.569900 26782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:36.571225 26767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:28:36.571419 26767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "9c197defe3f74743827d1c6aae65bbf5"
format_stamp: "Formatted at 2026-09-25 10:28:36 on dist-test-slave-f9rl"
server_key: "b321d59db0df2d9753efd53a1accf0cc"
server_key_iv: "abd3a3242cccf9cef367c52c5d9ca6de"
server_key_version: "encryptionkey@0"
I20260925 10:28:36.571637 26767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:36.591401 26767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:36.596243 26767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:36.596560 26767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:36.616386 26767 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:38161
I20260925 10:28:36.616410 26833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:38161 every 8 connection(s)
I20260925 10:28:36.617973 26767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:36.621773 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:36.623791 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26767
I20260925 10:28:36.624017 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:28:36.624673 13281 external_mini_cluster.cc:1428] Setting key 990bffb79af507bd79c5ff1030e6dae6
I20260925 10:28:36.628513 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:28:36 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.125@KRBTEST.COM" created.
I20260925 10:28:36.653683 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.631853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38161 (local address 127.0.0.1:34250)
0925 10:28:36.632727 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.632778 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:36.634976 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:36.635950 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:36.635955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:36.636511 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:36.636848 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:36.636865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:36.639274 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:36.639279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:36.640517 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:36.640530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:36.641950 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:36.641955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:36.642083 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:36.642785 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:36.642807 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:36.645154 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:36.649132 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:36.649142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:36.649152 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:36.649667 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:36.650105 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:36.650110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:36.650113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:36.650266 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:36.651080 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:36.651094 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:36.651304 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 10:28:36.651579 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 10:28:36.652220 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":156,"thread_start_us":381,"threads_started":1}
I20260925 10:28:36.653707 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.633195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.12.248.126:38161)
0925 10:28:36.634006 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:36.634019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:36.634073 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:28:36.635373 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:36.635384 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:36.635700 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:36.635915 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:36.637048 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:36.639037 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:36.640666 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:36.641731 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:36.641933 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:36.645408 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:36.645418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:36.645458 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:36.645514 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:36.648944 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:36.649802 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:36.649811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:36.649815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:36.649974 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:36.650371 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:36.650378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:36.650381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:36.650894 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:36.651145 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:36.651685 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 10:28:36.652114 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":579,"thread_start_us":347,"threads_started":1}
I20260925 10:28:36.657298 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:28:36.658514 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.658093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33710)
0925 10:28:36.658268 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.658381 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 10:28:36.658792 26785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
I20260925 10:28:36.660575 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.660081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33722)
0925 10:28:36.660278 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.660375 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:28:36.660802 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:34973: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
I20260925 10:28:36.677780 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:34973
--webserver_interface=127.12.248.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:36.710244 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } attempt: 1
I20260925 10:28:36.711617 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.711067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33728)
0925 10:28:36.711288 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.711444 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 10:28:36.712662 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.712243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33730)
0925 10:28:36.712400 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.712487 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 10:28:36.712872 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:34973: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
I20260925 10:28:36.781178 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } attempt: 2
I20260925 10:28:36.782548 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.781944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33734)
0925 10:28:36.782218 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.782332 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 10:28:36.783694 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.783209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33746)
0925 10:28:36.783387 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.783489 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:28:36.783903 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:34973: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
I20260925 10:28:36.875185 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } attempt: 3
I20260925 10:28:36.876520 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.875961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33754)
0925 10:28:36.876200 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.876329 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 10:28:36.877678 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:36.877230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33770)
0925 10:28:36.877384 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:36.877511 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:28:36.877934 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:34973: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
W20260925 10:28:37.009976 26841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:37.010279 26841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:37.010464 26841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:37.017706 26841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:37.017824 26841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:37.017943 26841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:37.018064 26841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:37.018131 26841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:37.021217 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } attempt: 4
I20260925 10:28:37.022514 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.021877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33782)
0925 10:28:37.022216 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.022331 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260925 10:28:37.023636 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.023199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33798)
0925 10:28:37.023358 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.023470 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:28:37.023837 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:34973: Network error: Client connection negotiation failed: client connection to 127.12.248.125:34973: connect: Connection refused (error 111)
I20260925 10:28:37.028959 26841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.248.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:37.030406 26841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:37.031786 26841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:37.043948 26846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:37.043949 26849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:37.044064 26847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:37.044711 26841 server_base.cc:1034] running on GCE node
I20260925 10:28:37.045576 26841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:37.047009 26841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:37.048213 26841 hybrid_clock.cc:648] HybridClock initialized: now 1790332117048172 us; error 28 us; skew 500 ppm
I20260925 10:28:37.048527 26841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:37.051819 26841 init.cc:377] Logged in from keytab as kudu/127.12.248.125@KRBTEST.COM (short username kudu)
I20260925 10:28:37.053323 26841 webserver.cc:460] Webserver started at http://127.12.248.125:46189/ using document root <none> and password file <none>
I20260925 10:28:37.054052 26841 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:37.054152 26841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:37.054402 26841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:37.057490 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data/instance:
uuid: "d42e53a0356442c484dbd8fa2cc7b908"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "b1c6bbdee3aa3a1e740822ae1c5bbe18"
server_key_iv: "a68b27b3136d6488781d32c2e6d394af"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.058277 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal/instance:
uuid: "d42e53a0356442c484dbd8fa2cc7b908"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "b1c6bbdee3aa3a1e740822ae1c5bbe18"
server_key_iv: "a68b27b3136d6488781d32c2e6d394af"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.063370 26841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:28:37.066790 26856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:37.068146 26841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:28:37.068320 26841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "d42e53a0356442c484dbd8fa2cc7b908"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "b1c6bbdee3aa3a1e740822ae1c5bbe18"
server_key_iv: "a68b27b3136d6488781d32c2e6d394af"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.068498 26841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:37.100046 26841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:37.104476 26841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:37.104775 26841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:37.123855 26841 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.125:34973
I20260925 10:28:37.123886 26907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.125:34973 every 8 connection(s)
I20260925 10:28:37.125229 26841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
I20260925 10:28:37.128988 26908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:37.130239 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26841
I20260925 10:28:37.130435 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-1/wal/instance
I20260925 10:28:37.131069 13281 external_mini_cluster.cc:1428] Setting key 9bec91f4c98010345e22088436719432
I20260925 10:28:37.139457 26908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.248.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.124@KRBTEST.COM" created.
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:37.162640 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.142614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.12.248.126:38161)
0925 10:28:37.142839 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.142846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.144128 (+ 1282us) server_negotiation.cc:407] Connection header received
0925 10:28:37.146125 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.146134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.146233 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.146353 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.148253 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.148935 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.150692 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.151122 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.151263 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.156618 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.156623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.156627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.156663 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.159392 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.160211 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.160216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.160226 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.160347 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.160808 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.160812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.160814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.161108 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.161270 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.162065 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.162299 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:28:37.163617 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.143087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38161 (local address 127.0.0.1:34258)
0925 10:28:37.143940 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.143991 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.145832 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.146576 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.146585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.147184 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.147970 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.148014 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.149114 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.149120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.150536 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.150549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.151356 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.151364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.151608 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.152304 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.152326 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.156344 (+ 4018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.159582 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.159591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.159608 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.160078 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.160483 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.160488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.160492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.160676 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.161241 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.161249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.161684 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.161953 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.162480 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":386,"threads_started":1}
I20260925 10:28:37.165325 26908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:28:37.180585 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.166196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33804)
0925 10:28:37.166354 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.166370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.166452 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.167615 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.167619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.168090 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.168220 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.168228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.169624 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.169631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.170453 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.170463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.171278 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.171283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.171540 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.172192 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.172209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.174575 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.178089 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.178095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.178099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.178446 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.178831 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.178837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.178842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.178999 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.179589 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.179594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.179723 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.179872 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.180162 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":4}
I20260925 10:28:37.180647 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.166505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.12.248.125:34973)
0925 10:28:37.167177 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.167182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.167208 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:28:37.167253 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.167259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.167437 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.167631 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.168373 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.169425 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.170589 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.171134 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.171268 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.174780 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.174791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.174803 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.174830 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.177958 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.178575 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.178590 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.178593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.178721 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.179101 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.179108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.179111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.179442 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.179664 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.180011 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.180263 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":533,"thread_start_us":391,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
I20260925 10:28:37.183218 26908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:28:37.184252 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.183801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49788)
0925 10:28:37.184011 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.184108 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 10:28:37.184455 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:39467
--webserver_interface=127.12.248.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:39359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:37.184597 26858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.186159 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.185826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49800)
0925 10:28:37.185978 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.186065 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:28:37.186456 26908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.235797 26908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 1
I20260925 10:28:37.236956 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.236503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49802)
0925 10:28:37.236699 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.236807 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:28:37.238204 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.237754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49818)
0925 10:28:37.237968 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.238056 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 10:28:37.238438 26908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.306710 26908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 2
I20260925 10:28:37.308055 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.307492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49830)
0925 10:28:37.307740 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.307861 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 10:28:37.309099 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.308734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49832)
0925 10:28:37.308877 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.308958 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:28:37.309309 26908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.323158 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } attempt: 5
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:28:37.338385 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.323949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33814)
0925 10:28:37.324191 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.324214 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.324347 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.324750 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.324754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.325141 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.325317 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.325326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.326844 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.326848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.327709 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.327720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.328900 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.328904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.329045 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.329775 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.329793 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.332073 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.335939 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.335943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.335946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.336333 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.336804 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.336807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.336810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.336940 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.337633 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.337637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.337770 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.337887 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.338138 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 10:28:37.338559 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.324185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.12.248.125:34973)
0925 10:28:37.324398 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.324404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.324427 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:28:37.324522 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.324528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.324606 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.324714 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.325515 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.326699 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.327871 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.328743 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.328908 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.332287 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.332294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.332299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.332342 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.335772 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.336472 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.336480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.336486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.336676 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.337072 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.337078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.337083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.337525 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.337683 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.338023 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.338250 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:28:37.339634 26834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:28:37.340620 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.340261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49846)
0925 10:28:37.340421 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.340500 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 10:28:37.341590 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.341183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49862)
0925 10:28:37.341324 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.341409 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:28:37.341809 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.393059 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 1
I20260925 10:28:37.394429 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.393765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49870)
0925 10:28:37.394120 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.394232 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260925 10:28:37.396024 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.395119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49880)
0925 10:28:37.395644 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.395803 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
W20260925 10:28:37.396325 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.400530 26908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 3
I20260925 10:28:37.401767 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.401213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49896)
0925 10:28:37.401481 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.401591 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 10:28:37.402963 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.402452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49908)
0925 10:28:37.402753 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.402836 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
W20260925 10:28:37.403275 26908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.464535 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 2
I20260925 10:28:37.465790 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.465297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49918)
0925 10:28:37.465557 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.465649 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 10:28:37.466840 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.466486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49928)
0925 10:28:37.466646 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.466721 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:28:37.467042 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
W20260925 10:28:37.539954 26915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:37.540390 26915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:37.540583 26915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:28:37.546499 26908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 4
I20260925 10:28:37.547760 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.547202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49934)
0925 10:28:37.547458 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.547576 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:28:37.548740 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.548395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49946)
0925 10:28:37.548538 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.548617 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 10:28:37.548942 26908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
W20260925 10:28:37.549052 26915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:37.549186 26915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:37.549278 26915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:37.549371 26915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:37.549453 26915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:37.559957 26915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:39359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38161,127.12.248.125:34973,127.12.248.124:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.248.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:37.561363 26915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:37.562768 26915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:28:37.573225 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 3
I20260925 10:28:37.574599 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.573961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49948)
0925 10:28:37.574164 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.574353 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 10:28:37.575208 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:37.575572 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:37.576037 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.575469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49964)
0925 10:28:37.575758 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.575856 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260925 10:28:37.576275 26834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:39467: Network error: Client connection negotiation failed: client connection to 127.12.248.124:39467: connect: Connection refused (error 111)
I20260925 10:28:37.576401 26915 server_base.cc:1034] running on GCE node
W20260925 10:28:37.576756 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:37.577745 26915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:37.579187 26915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:37.580428 26915 hybrid_clock.cc:648] HybridClock initialized: now 1790332117580393 us; error 45 us; skew 500 ppm
I20260925 10:28:37.580786 26915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:37.583933 26915 init.cc:377] Logged in from keytab as kudu/127.12.248.124@KRBTEST.COM (short username kudu)
I20260925 10:28:37.585418 26915 webserver.cc:460] Webserver started at http://127.12.248.124:35193/ using document root <none> and password file <none>
I20260925 10:28:37.586160 26915 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:37.586261 26915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:37.586534 26915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:37.589506 26915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data/instance:
uuid: "c4cbd77618d44a00b6d7b333c00c9061"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "2e95a72c43dff038fbf06d3792db4203"
server_key_iv: "36e6eb36e63fd3978a4b3b1c3bfc0bfa"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.590269 26915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal/instance:
uuid: "c4cbd77618d44a00b6d7b333c00c9061"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "2e95a72c43dff038fbf06d3792db4203"
server_key_iv: "36e6eb36e63fd3978a4b3b1c3bfc0bfa"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.595224 26915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 10:28:37.598508 26931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:37.599846 26915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:28:37.599979 26915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "c4cbd77618d44a00b6d7b333c00c9061"
format_stamp: "Formatted at 2026-09-25 10:28:37 on dist-test-slave-f9rl"
server_key: "2e95a72c43dff038fbf06d3792db4203"
server_key_iv: "36e6eb36e63fd3978a4b3b1c3bfc0bfa"
server_key_version: "encryptionkey@0"
I20260925 10:28:37.600137 26915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:37.639796 26915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:37.644033 26915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:37.644275 26915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:37.662770 26915 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.124:39467
I20260925 10:28:37.662803 26982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.124:39467 every 8 connection(s)
I20260925 10:28:37.664206 26915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
I20260925 10:28:37.667855 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 26915
I20260925 10:28:37.668089 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331906218802-13281-0/minicluster-data/master-2/wal/instance
I20260925 10:28:37.668183 26983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:37.668686 13281 external_mini_cluster.cc:1428] Setting key 04bf8d0669f5da12d1da471db8f16829
I20260925 10:28:37.678406 26983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:37.692268 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.670632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.12.248.124:39467)
0925 10:28:37.671477 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.671488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.671529 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:28:37.673263 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.673285 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.673767 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.674089 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.675896 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.677389 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.679388 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.680085 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.680218 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.684866 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.684875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.684884 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.684932 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.688142 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.688962 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.688967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.688972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.689133 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.689570 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.689576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.689581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.690157 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.690454 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.690907 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.691303 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":416,"threads_started":1}
I20260925 10:28:37.696072 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.681356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.12.248.126:38161)
0925 10:28:37.681976 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.681981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.682407 (+ 426us) server_negotiation.cc:407] Connection header received
0925 10:28:37.682609 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.682621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.682685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.682801 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.683846 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.684682 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.686351 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.687083 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.687220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.690540 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.690545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.690548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.690579 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.693262 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.693998 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.694002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.694004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.694124 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.694542 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.694547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.694552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.694875 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.695040 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.695578 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.695753 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":352,"threads_started":1}
I20260925 10:28:37.696427 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.681377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38161 (local address 127.0.0.1:34266)
0925 10:28:37.682278 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.682312 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.682482 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.682864 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.682871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.683449 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.683694 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.683702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.684851 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.684855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.686203 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.686219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.687277 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.687281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.687405 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.687976 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.687999 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.690351 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.693393 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.693405 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.693412 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.693854 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.694238 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.694243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.694247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.694402 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.695037 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.695046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.695232 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.695473 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.695897 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":554,"threads_started":1}
I20260925 10:28:37.698391 26983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:28:37.712292 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.699127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:34973 (local address 127.0.0.1:33822)
0925 10:28:37.699292 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.699310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.699394 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.699653 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.699656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.700116 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.700300 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.700312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.701236 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.701244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.702470 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.702482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.703201 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.703208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.703471 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.703953 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.703980 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.706950 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.709923 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.709926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.709929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.710273 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.710678 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.710682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.710686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.710820 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.711320 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.711325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.711507 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.711671 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.711941 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:28:37.712312 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.695181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.12.248.126:38161)
0925 10:28:37.696318 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.696323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.696339 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:28:37.696385 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.696390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.696510 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.696628 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.698484 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.699782 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.701303 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.701762 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.701854 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.706885 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.706891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.706895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.706933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.709748 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.710423 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.710426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.710428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.710539 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.710906 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.710913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.710916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.711203 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.711474 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.711751 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.711959 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":1353,"threads_started":1}
I20260925 10:28:37.712438 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.699241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.12.248.125:34973)
0925 10:28:37.699396 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.699400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.699417 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:28:37.699482 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.699486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.699545 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.699617 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.700446 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.701095 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.702634 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.703051 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.703143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.707145 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.707150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.707153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.707186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.709770 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.710389 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.710397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.710404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.710547 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.710949 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.710953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.710955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.711202 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.711360 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.711785 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.712029 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:28:37.713366 26983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:28:37.727265 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.714159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.12.248.124:39467)
0925 10:28:37.714317 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.714322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.714469 (+ 147us) server_negotiation.cc:407] Connection header received
0925 10:28:37.714642 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.714647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.714707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.714800 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.715419 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.716081 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.717167 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.717668 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.717803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.721120 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.721126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.721130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.721165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.724465 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.725202 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.725208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.725212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.725347 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.725763 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.725768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.725772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.726167 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.726325 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.726662 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.727003 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":55}
I20260925 10:28:37.727334 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.714104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49976)
0925 10:28:37.714381 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.714397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.714533 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.714816 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.714819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.715174 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.715293 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.715300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.716202 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.716208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.717037 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.717047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.717820 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.717825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.718025 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.718605 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.718628 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.720929 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.724605 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.724609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.724611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.725085 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.725521 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.725524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.725527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.725649 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.726290 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.726293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.726410 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.726538 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.726875 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260925 10:28:37.729533 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.714206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.12.248.125:34973)
0925 10:28:37.714346 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.714349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.714550 (+ 201us) server_negotiation.cc:407] Connection header received
0925 10:28:37.714715 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.714720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.714788 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.714883 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.716045 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.716729 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.718046 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.718503 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.718628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.723154 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.723162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.723167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.723216 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.726501 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.727370 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.727377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.727382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.727559 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.728017 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.728022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.728026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.728287 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.728482 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.728895 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.729134 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:28:37.733072 26983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: Bootstrap starting.
I20260925 10:28:37.734618 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:28:37.736469 26983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:37.737629 26983 log.cc:824] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:37.740895 26983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: No bootstrap required, opened a new log
I20260925 10:28:37.753535 26834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 4
I20260925 10:28:37.756265 26983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.756992 26983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:37.757210 26983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cbd77618d44a00b6d7b333c00c9061, State: Initialized, Role: FOLLOWER
I20260925 10:28:37.758313 26983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.763813 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.739585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.12.248.126:38161)
0925 10:28:37.739773 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.739779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.740382 (+ 603us) server_negotiation.cc:407] Connection header received
0925 10:28:37.740747 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.740756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.740860 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.740964 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.744097 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.745908 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.749828 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.750556 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.750671 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.754706 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.754713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.754718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.754757 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.760317 (+ 5560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.761191 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.761199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.761203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.761369 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.761905 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.761911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.761914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.762306 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.762528 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.763160 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.763412 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:28:37.766506 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.740111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.12.248.125:34973)
0925 10:28:37.740279 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.740285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.748569 (+ 8284us) server_negotiation.cc:407] Connection header received
0925 10:28:37.748764 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.748797 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.748893 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.748997 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.751448 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.752607 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.754897 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.755616 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.755740 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.759339 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.759346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.759351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.759396 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.763077 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.764003 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.764009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.764013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.764160 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.764653 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.764658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.764661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.765031 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.765257 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.765915 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.766168 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:28:37.767674 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:37.768236 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:37.769390 26983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332116, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:28:37.776069 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.740809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.12.248.124:39467)
0925 10:28:37.741775 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.741780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.752121 (+ 10341us) server_negotiation.cc:407] Connection header received
0925 10:28:37.752546 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.752552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.752634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.752700 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.760575 (+ 7875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.761851 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.764337 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.765021 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.765116 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.768724 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.768732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.768739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.768787 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.772260 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.773171 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.773180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.773183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.773334 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.773785 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.773789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.773793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.774200 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.774530 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.775127 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.775486 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828}
I20260925 10:28:37.780591 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.754078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49988)
0925 10:28:37.754439 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.754456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.754559 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.759999 (+ 5440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.760007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.760588 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.760892 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.760904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.764298 (+ 3394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.764307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.765836 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.765852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.768567 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.768576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.768781 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.769378 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.769403 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.772505 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.777367 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.777376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.777383 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.778087 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.778597 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.778602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.778605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.778785 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.779484 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.779488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.779648 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.779849 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.780174 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260925 10:28:37.780711 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.754757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.12.248.124:39467)
0925 10:28:37.759529 (+ 4772us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.759536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.759562 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:28:37.759616 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.759623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.759698 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.761062 (+ 1364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.761342 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.762521 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.766079 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.766777 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.771178 (+ 4401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.772810 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.772817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.772822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.772862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.777190 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.778280 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.778284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.778290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.778439 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.778938 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.778946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.778949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.779339 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.779516 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.779987 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.780197 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4622,"thread_start_us":4436,"threads_started":1}
I20260925 10:28:37.786872 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: Bootstrap starting.
I20260925 10:28:37.790850 26834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:37.791868 26834 log.cc:824] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:37.780645 26915 master_runner.cc:186] Error getting master registration for 127.12.248.126:38161: OK, instance_id { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" instance_seqno: 1790332116610091 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:28:37.793385 26915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:28:37.794380 27010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:28:37.794509 27010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:28:37.796644 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: No bootstrap required, opened a new log
I20260925 10:28:37.804595 26834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.805074 26834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:37.805243 26834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c197defe3f74743827d1c6aae65bbf5, State: Initialized, Role: FOLLOWER
I20260925 10:28:37.806063 26834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.807055 27013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:37.807437 27013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:37.808249 26834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:28:37.819675 27024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:28:37.819911 27024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:28:37.819204 26767 master_runner.cc:186] Error getting master registration for 127.12.248.125:34973: OK, instance_id { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" instance_seqno: 1790332117118593 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:28:37.832746 26767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:28:37.848181 26908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } attempt: 5
W20260925 10:28:37.849633 26788 tablet.cc:2369] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:28:37 dist-test-slave-f9rl krb5kdc[26754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:28:37.868541 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.849708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.12.248.124:39467)
0925 10:28:37.849909 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:37.849915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:37.850280 (+ 365us) server_negotiation.cc:407] Connection header received
0925 10:28:37.850345 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:37.850354 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:37.850490 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:37.850587 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:37.852046 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.853203 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.855073 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.855769 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.855907 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.860033 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:37.860040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:37.860045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:37.860086 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:37.864101 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.865018 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.865024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.865028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.865187 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:37.865699 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:37.865704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:37.865708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:37.866105 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:37.866249 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:37.867912 (+ 1663us) server_negotiation.cc:299] Negotiation successful
0925 10:28:37.868164 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:28:37.868841 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:37.848928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:39467 (local address 127.0.0.1:49994)
0925 10:28:37.850029 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:37.850050 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:37.850171 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:37.851107 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:37.851114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:37.851657 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:37.851879 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.851892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.853363 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.853369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.854891 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:37.854907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:37.855949 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:37.855956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:37.856195 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:37.856924 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:37.856948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:37.859817 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:37.864277 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.864283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.864288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.864865 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.865342 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:37.865348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:37.865352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:37.865559 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:37.866320 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:37.866325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:37.866489 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 10:28:37.866649 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 10:28:37.868283 (+ 1634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447}
I20260925 10:28:37.875131 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: Bootstrap starting.
I20260925 10:28:37.879597 26908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:37.882109 26908 log.cc:824] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:37.886094 26908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: No bootstrap required, opened a new log
W20260925 10:28:37.896452 26937 tablet.cc:2369] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:28:37.897315 26908 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.897776 26908 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:37.897907 26908 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e53a0356442c484dbd8fa2cc7b908, State: Initialized, Role: FOLLOWER
I20260925 10:28:37.898594 26908 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.903136 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:37.903725 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:37.904600 26908 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:37.930203 26841 master_runner.cc:263] No leader master found from master d42e53a0356442c484dbd8fa2cc7b908
I20260925 10:28:37.930370 26841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:28:37.931927 27038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:28:37.932066 27038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:28:37.992986 27027 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:37.993363 27027 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:37.995111 27027 leader_election.cc:288] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c197defe3f74743827d1c6aae65bbf5 (127.12.248.126:38161), c4cbd77618d44a00b6d7b333c00c9061 (127.12.248.124:39467)
I20260925 10:28:37.996914 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d42e53a0356442c484dbd8fa2cc7b908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c197defe3f74743827d1c6aae65bbf5" is_pre_election: true
I20260925 10:28:37.997593 26809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42e53a0356442c484dbd8fa2cc7b908 in term 0.
I20260925 10:28:37.998704 26958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d42e53a0356442c484dbd8fa2cc7b908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cbd77618d44a00b6d7b333c00c9061" is_pre_election: true
I20260925 10:28:37.999306 26958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d42e53a0356442c484dbd8fa2cc7b908 in term 0.
I20260925 10:28:38.000075 26858 leader_election.cc:302] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4cbd77618d44a00b6d7b333c00c9061, d42e53a0356442c484dbd8fa2cc7b908; no voters:
I20260925 10:28:38.000793 27027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:28:38.000937 27027 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:28:38.001031 27027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:38.006307 27027 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:38.010025 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d42e53a0356442c484dbd8fa2cc7b908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c197defe3f74743827d1c6aae65bbf5"
I20260925 10:28:38.010319 26809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:38.014297 26809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42e53a0356442c484dbd8fa2cc7b908 in term 1.
I20260925 10:28:38.015110 26958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d42e53a0356442c484dbd8fa2cc7b908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cbd77618d44a00b6d7b333c00c9061"
I20260925 10:28:38.015365 26958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:38.019001 26958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d42e53a0356442c484dbd8fa2cc7b908 in term 1.
I20260925 10:28:38.019701 26858 leader_election.cc:302] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4cbd77618d44a00b6d7b333c00c9061, d42e53a0356442c484dbd8fa2cc7b908; no voters:
I20260925 10:28:38.020473 27027 leader_election.cc:288] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [CANDIDATE]: Term 1 election: Requested vote from peers 9c197defe3f74743827d1c6aae65bbf5 (127.12.248.126:38161), c4cbd77618d44a00b6d7b333c00c9061 (127.12.248.124:39467)
I20260925 10:28:38.020772 27027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:38.020972 27027 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [term 1 LEADER]: Becoming Leader. State: Replica: d42e53a0356442c484dbd8fa2cc7b908, State: Running, Role: LEADER
I20260925 10:28:38.021651 27027 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } }
I20260925 10:28:38.039129 27040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e53a0356442c484dbd8fa2cc7b908. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.039721 27040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:38.045517 27042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:38.048014 27042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:38.067574 26958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [term 1 FOLLOWER]: Refusing update from remote peer d42e53a0356442c484dbd8fa2cc7b908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:38.068733 27040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:38.072911 26809 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [term 1 FOLLOWER]: Refusing update from remote peer d42e53a0356442c484dbd8fa2cc7b908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:38.074025 27040 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:38.098575 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e53a0356442c484dbd8fa2cc7b908. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.098934 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:38.099473 27043 mvcc.cc:205] Tried to move back new op lower bound from 7333200355592429568 to 7333200355427340288. Current Snapshot: MvccSnapshot[applied={T|T < 7333200355592429568}]
I20260925 10:28:38.118790 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.119103 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:38.125520 27042 catalog_manager.cc:1347] Generated new cluster ID: d50d9503273242beacea84fac55c8fcb
I20260925 10:28:38.125655 27042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:38.130012 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.131769 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:38.150221 27013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e53a0356442c484dbd8fa2cc7b908. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.150530 27013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:38.151849 27040 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.152084 27040 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:38.153707 27045 mvcc.cc:205] Tried to move back new op lower bound from 7333200355592429568 to 7333200355427340288. Current Snapshot: MvccSnapshot[applied={T|T < 7333200355592429568}]
I20260925 10:28:38.165904 27042 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:28:38.168793 27042 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:38.175305 27013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d42e53a0356442c484dbd8fa2cc7b908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c197defe3f74743827d1c6aae65bbf5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38161 } } peers { permanent_uuid: "d42e53a0356442c484dbd8fa2cc7b908" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 34973 } } peers { permanent_uuid: "c4cbd77618d44a00b6d7b333c00c9061" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 39467 } } }
I20260925 10:28:38.175581 27013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:28:38.181476 27042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42e53a0356442c484dbd8fa2cc7b908: Generated new TSK 0
I20260925 10:28:38.182600 27042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:28:38.796526 27010 catalog_manager.cc:1259] Loaded cluster ID: d50d9503273242beacea84fac55c8fcb
I20260925 10:28:38.796638 27010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: loading cluster ID for follower catalog manager: success
I20260925 10:28:38.799024 27010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: acquiring CA information for follower catalog manager: success
I20260925 10:28:38.800770 27010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4cbd77618d44a00b6d7b333c00c9061: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:28:38.821938 27024 catalog_manager.cc:1259] Loaded cluster ID: d50d9503273242beacea84fac55c8fcb
I20260925 10:28:38.822067 27024 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: loading cluster ID for follower catalog manager: success
I20260925 10:28:38.824434 27024 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: acquiring CA information for follower catalog manager: success
I20260925 10:28:38.826236 27024 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c197defe3f74743827d1c6aae65bbf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:28:39.322257 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26767
I20260925 10:28:39.323846 26829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:39.435250 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26767
I20260925 10:28:39.456326 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26841
I20260925 10:28:39.457584 26903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:39.568641 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26841
I20260925 10:28:39.589722 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 26915
I20260925 10:28:39.590968 26978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:39.700342 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 26915
2026-09-25T10:28:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4922 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:28:39 dist-test-slave-f9rl krb5kdc[27061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:28:39 dist-test-slave-f9rl krb5kdc[27061](info): set up 2 sockets
Sep 25 10:28:39 dist-test-slave-f9rl krb5kdc[27061](info): commencing operation
krb5kdc: starting...
W20260925 10:28:40.815341 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:28:40 dist-test-slave-f9rl krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:28:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:28:41.057341 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:33035
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:41.397780 27074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:41.398080 27074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:41.398252 27074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:41.405493 27074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:41.405645 27074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:41.405740 27074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:41.405839 27074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:41.405926 27074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:41.415977 27074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:41.417457 27074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:41.418803 27074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:41.430932 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:41.431002 27080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:41.431041 27082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:41.432041 27074 server_base.cc:1034] running on GCE node
I20260925 10:28:41.432884 27074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:41.434458 27074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:41.435730 27074 hybrid_clock.cc:648] HybridClock initialized: now 1790332121435676 us; error 35 us; skew 500 ppm
I20260925 10:28:41.436156 27074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:41 dist-test-slave-f9rl krb5kdc[27061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332121, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:41.439684 27074 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:41.441399 27074 webserver.cc:460] Webserver started at http://127.12.248.126:33697/ using document root <none> and password file <none>
I20260925 10:28:41.442245 27074 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:41.442371 27074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:41.442655 27074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:41.445950 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "c388999caa704cf3aeec448647b2f44c"
format_stamp: "Formatted at 2026-09-25 10:28:41 on dist-test-slave-f9rl"
server_key: "f95c40213588c8a7f49350769f7e6f90"
server_key_iv: "2d2beb62d9ac1bde63d64f1712276ec8"
server_key_version: "encryptionkey@0"
I20260925 10:28:41.446784 27074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "c388999caa704cf3aeec448647b2f44c"
format_stamp: "Formatted at 2026-09-25 10:28:41 on dist-test-slave-f9rl"
server_key: "f95c40213588c8a7f49350769f7e6f90"
server_key_iv: "2d2beb62d9ac1bde63d64f1712276ec8"
server_key_version: "encryptionkey@0"
I20260925 10:28:41.452693 27074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 10:28:41.456204 27089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:41.457625 27074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:41.457772 27074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "c388999caa704cf3aeec448647b2f44c"
format_stamp: "Formatted at 2026-09-25 10:28:41 on dist-test-slave-f9rl"
server_key: "f95c40213588c8a7f49350769f7e6f90"
server_key_iv: "2d2beb62d9ac1bde63d64f1712276ec8"
server_key_version: "encryptionkey@0"
I20260925 10:28:41.457947 27074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:41.485126 27074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:41.489622 27074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:41.489883 27074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:41.508632 27074 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:33035
I20260925 10:28:41.508663 27140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:33035 every 8 connection(s)
I20260925 10:28:41.510056 27074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:41.513953 27141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:41.520372 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27074
I20260925 10:28:41.520622 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:28:41.521313 13281 external_mini_cluster.cc:1428] Setting key d3766a0b1fa2e28ddeb97a5cb55445ba
I20260925 10:28:41.525135 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c: Bootstrap starting.
I20260925 10:28:41.529021 27141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:41.530133 27141 log.cc:824] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:41.533209 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c: No bootstrap required, opened a new log
Sep 25 10:28:41 dist-test-slave-f9rl krb5kdc[27061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:41.541782 27141 raft_consensus.cc:348] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } }
I20260925 10:28:41.542237 27141 raft_consensus.cc:374] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:41.542349 27141 raft_consensus.cc:729] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c388999caa704cf3aeec448647b2f44c, State: Initialized, Role: FOLLOWER
I20260925 10:28:41.543036 27141 consensus_queue.cc:260] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } }
I20260925 10:28:41.543373 27141 raft_consensus.cc:388] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:41.543541 27141 raft_consensus.cc:482] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:41.543759 27141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:41.545116 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:41.523347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.12.248.126:33035)
0925 10:28:41.524149 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:41.524161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:41.524201 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:28:41.526123 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:41.526146 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:41.526544 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:41.526884 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:41.528745 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:41.530200 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:41.531918 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:41.532564 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:41.532697 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:41.537306 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:41.537314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:41.537328 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:41.537372 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:41.540604 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:41.541384 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:41.541392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:41.541398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:41.541644 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:41.542068 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:41.542073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:41.542078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:41.542765 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:41.543079 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:41.543638 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 10:28:41.544072 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260925 10:28:41.546373 27141 raft_consensus.cc:504] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } }
I20260925 10:28:41.547173 27141 leader_election.cc:302] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c388999caa704cf3aeec448647b2f44c; no voters:
I20260925 10:28:41.547863 27141 leader_election.cc:288] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:28:41.547982 27146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:41.549125 27146 raft_consensus.cc:686] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [term 1 LEADER]: Becoming Leader. State: Replica: c388999caa704cf3aeec448647b2f44c, State: Running, Role: LEADER
I20260925 10:28:41.549362 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:28:41.549765 27146 consensus_queue.cc:237] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } }
I20260925 10:28:41.550263 27141 sys_catalog.cc:564] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:41.552762 27148 sys_catalog.cc:455] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c388999caa704cf3aeec448647b2f44c. Latest consensus state: current_term: 1 leader_uuid: "c388999caa704cf3aeec448647b2f44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } } }
I20260925 10:28:41.553236 27148 sys_catalog.cc:458] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:41.553639 27147 sys_catalog.cc:455] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c388999caa704cf3aeec448647b2f44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c388999caa704cf3aeec448647b2f44c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 33035 } } }
I20260925 10:28:41.553951 27147 sys_catalog.cc:458] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:41.558115 27152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:41.563499 27152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:41.572266 27152 catalog_manager.cc:1347] Generated new cluster ID: e9084465ae6f462193e0d5f82a6f059c
I20260925 10:28:41.572399 27152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:41.598028 27152 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:28:41.599535 27152 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:41.607461 27152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c388999caa704cf3aeec448647b2f44c: Generated new TSK 0
I20260925 10:28:41.608315 27152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:28:41.716040 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27074
I20260925 10:28:41.717614 27136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:41.819597 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27074
2026-09-25T10:28:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2114 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:28:41 dist-test-slave-f9rl krb5kdc[27169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:28:41 dist-test-slave-f9rl krb5kdc[27169](info): set up 2 sockets
Sep 25 10:28:41 dist-test-slave-f9rl krb5kdc[27169](info): commencing operation
krb5kdc: starting...
W20260925 10:28:42.983668 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:28:43 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:28:43.232926 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:43.594748 27182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:43.595120 27182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:43.595309 27182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:43.602824 27182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:43.602991 27182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:43.603077 27182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:43.603184 27182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:43.603269 27182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:43.613735 27182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:46803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:43.615201 27182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:43.616623 27182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:43.630319 27187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:43.630262 27190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:43.630292 27188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:43.631110 27182 server_base.cc:1034] running on GCE node
I20260925 10:28:43.631880 27182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:43.633410 27182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:43.634708 27182 hybrid_clock.cc:648] HybridClock initialized: now 1790332123634638 us; error 54 us; skew 500 ppm
I20260925 10:28:43.635083 27182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:43 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:43.638527 27182 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:43.640820 27182 webserver.cc:460] Webserver started at http://127.12.248.126:42387/ using document root <none> and password file <none>
I20260925 10:28:43.641789 27182 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:43.641975 27182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:43.642373 27182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:43.645756 27182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "65ae6424aad74bf9a68f489b6f26f399"
format_stamp: "Formatted at 2026-09-25 10:28:43 on dist-test-slave-f9rl"
server_key: "46fe916f70e21327ce4bad8d425d9d5e"
server_key_iv: "bc464c0c0666bde366efdf3c534e3b0d"
server_key_version: "encryptionkey@0"
I20260925 10:28:43.646616 27182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "65ae6424aad74bf9a68f489b6f26f399"
format_stamp: "Formatted at 2026-09-25 10:28:43 on dist-test-slave-f9rl"
server_key: "46fe916f70e21327ce4bad8d425d9d5e"
server_key_iv: "bc464c0c0666bde366efdf3c534e3b0d"
server_key_version: "encryptionkey@0"
I20260925 10:28:43.652209 27182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:28:43.656042 27197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:43.657727 27182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:28:43.657908 27182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "65ae6424aad74bf9a68f489b6f26f399"
format_stamp: "Formatted at 2026-09-25 10:28:43 on dist-test-slave-f9rl"
server_key: "46fe916f70e21327ce4bad8d425d9d5e"
server_key_iv: "bc464c0c0666bde366efdf3c534e3b0d"
server_key_version: "encryptionkey@0"
I20260925 10:28:43.658107 27182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:43.671623 27182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:43.676470 27182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:43.676805 27182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:43.698827 27182 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:46803
I20260925 10:28:43.698846 27248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:46803 every 8 connection(s)
I20260925 10:28:43.700369 27182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:43.704210 27249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:43.706956 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27182
I20260925 10:28:43.707193 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:28:43.708216 13281 external_mini_cluster.cc:1428] Setting key 6cd4bb455ac8390de46187a76877b774
I20260925 10:28:43.718092 27249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap starting.
I20260925 10:28:43.722200 27249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:43.723614 27249 log.cc:824] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:28:43 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:43.726872 27249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: No bootstrap required, opened a new log
I20260925 10:28:43.733927 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:43.710529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.12.248.126:46803)
0925 10:28:43.711344 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:43.711356 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:43.711406 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:28:43.713175 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:43.713198 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:43.713700 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:43.714057 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:43.716471 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:43.717910 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:43.719833 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:43.720596 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:43.720735 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:43.726238 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:43.726246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:43.726260 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:43.726296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:43.729592 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:43.730323 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:43.730329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:43.730338 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:43.730506 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:43.730917 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:43.730926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:43.730928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:43.731488 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:43.731847 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:43.732327 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 10:28:43.732771 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":382,"threads_started":1}
I20260925 10:28:43.735569 27249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:43.735968 27249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:43.736081 27249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Initialized, Role: FOLLOWER
I20260925 10:28:43.736653 27249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:43.736922 27249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:43.737088 27249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:43.737254 27249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:43.739769 27249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:43.740268 27249 leader_election.cc:302] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ae6424aad74bf9a68f489b6f26f399; no voters:
I20260925 10:28:43.740965 27249 leader_election.cc:288] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:28:43.741080 27254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:43.741988 27254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 LEADER]: Becoming Leader. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Running, Role: LEADER
I20260925 10:28:43.742641 27254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:43.743175 27249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:43.745576 27256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ae6424aad74bf9a68f489b6f26f399. Latest consensus state: current_term: 1 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:43.745708 27255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:43.745994 27256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:43.746129 27255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:43.748334 27262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:43.753700 27262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:43.764235 27262 catalog_manager.cc:1347] Generated new cluster ID: 15b0e781d01c41eaaaad02b3a66a07cc
I20260925 10:28:43.764336 27262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:43.773881 27262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:28:43.775825 27262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:43.785750 27262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Generated new TSK 0
I20260925 10:28:43.786639 27262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:28:43.855003 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:44.229984 27275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:44.230261 27275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:44.230450 27275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:44.238041 27275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:44.238335 27275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:44.238580 27275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:28:44.249128 27275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:44.250500 27275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:44.251856 27275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:44.265671 27280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:44.265662 27283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:44.265730 27281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:44.266968 27275 server_base.cc:1034] running on GCE node
I20260925 10:28:44.267717 27275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:44.268743 27275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:44.270012 27275 hybrid_clock.cc:648] HybridClock initialized: now 1790332124269951 us; error 72 us; skew 500 ppm
I20260925 10:28:44.270445 27275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:44 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:44.274006 27275 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:28:44.275519 27275 webserver.cc:460] Webserver started at http://127.12.248.65:41941/ using document root <none> and password file <none>
I20260925 10:28:44.276180 27275 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:44.276285 27275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:44.276543 27275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:44.279471 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "2f32a89e2294427da7ebfd14dc257ec4"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "a4780896b7d1dece83782a8d7bd58df1"
server_key_iv: "5520e3d71781e1c326d88dc60b9ae53c"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.280267 27275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "2f32a89e2294427da7ebfd14dc257ec4"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "a4780896b7d1dece83782a8d7bd58df1"
server_key_iv: "5520e3d71781e1c326d88dc60b9ae53c"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.285647 27275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 10:28:44.289179 27290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.290649 27275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:44.290846 27275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "2f32a89e2294427da7ebfd14dc257ec4"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "a4780896b7d1dece83782a8d7bd58df1"
server_key_iv: "5520e3d71781e1c326d88dc60b9ae53c"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.291067 27275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:44.310899 27275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:44.315214 27275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:44.315524 27275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:44.316535 27275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:44.318063 27275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:44.318168 27275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.318326 27275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:44.318399 27275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.354187 27275 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36175
I20260925 10:28:44.354236 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36175 every 8 connection(s)
I20260925 10:28:44.355511 27275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:28:44.365528 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27275
I20260925 10:28:44.365767 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:28:44.366585 13281 external_mini_cluster.cc:1428] Setting key 8e5222bc9dfbf4e4a95200a751ffa7db
Sep 25 10:28:44 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:44.377283 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:44.358271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:57905 (local address 127.12.248.126:46803)
0925 10:28:44.358746 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:44.358751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:44.359648 (+ 897us) server_negotiation.cc:407] Connection header received
0925 10:28:44.361649 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:44.361654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:44.361757 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:44.361877 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:44.363526 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.364260 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.366518 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.367003 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.367119 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:44.371333 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:44.371399 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:44.371402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:44.371434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:44.374159 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:44.374908 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:44.374914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:44.374917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:44.375050 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:44.375504 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:44.375513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:44.375516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:44.375773 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:44.375938 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:44.376677 (+ 739us) server_negotiation.cc:299] Negotiation successful
0925 10:28:44.376906 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":260,"threads_started":1}
I20260925 10:28:44.378441 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:44.358678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:57905)
0925 10:28:44.359481 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:44.359530 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:44.361324 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:44.362161 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:44.362175 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:44.362766 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:44.363344 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:44.363361 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.364428 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.364433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:44.366335 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:44.366351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.367327 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.367332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:44.367619 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:44.368382 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:44.368412 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:44.371097 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:44.374313 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:44.374322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:44.374340 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:44.374770 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:44.375170 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:44.375174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:44.375178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:44.375380 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:44.375913 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:44.375920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:44.376294 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 10:28:44.376572 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 10:28:44.377135 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":290,"threads_started":1}
I20260925 10:28:44.380129 27405 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:44.380442 27405 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:44.381218 27405 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
I20260925 10:28:44.383801 27214 ts_manager.cc:194] Registered new tserver with Master: 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175)
I20260925 10:28:44.386128 27214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:57905
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:28:44.420706 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:44.758948 27411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:44.759239 27411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:44.759425 27411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:44.766963 27411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:44.767104 27411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:44.767299 27411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:28:44.777230 27411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:44.778612 27411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:44.779991 27411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:44.793964 27416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:44.794081 27417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:44.794186 27419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:44.795269 27411 server_base.cc:1034] running on GCE node
I20260925 10:28:44.795897 27411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:44.796913 27411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:44.798117 27411 hybrid_clock.cc:648] HybridClock initialized: now 1790332124798053 us; error 51 us; skew 500 ppm
I20260925 10:28:44.798444 27411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:44 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:44.801685 27411 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:28:44.803471 27411 webserver.cc:460] Webserver started at http://127.12.248.66:37761/ using document root <none> and password file <none>
I20260925 10:28:44.804152 27411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:44.804368 27411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:44.804772 27411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:44.807770 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "7b4ad05bbb32e23f0a676bec5f72a4c7"
server_key_iv: "401389b645ab97ee23d2cd6fe8156934"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.808573 27411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "7b4ad05bbb32e23f0a676bec5f72a4c7"
server_key_iv: "401389b645ab97ee23d2cd6fe8156934"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.814054 27411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:28:44.817878 27426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.819315 27411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:28:44.819554 27411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "7b4ad05bbb32e23f0a676bec5f72a4c7"
server_key_iv: "401389b645ab97ee23d2cd6fe8156934"
server_key_version: "encryptionkey@0"
I20260925 10:28:44.819789 27411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:44.837357 27411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:44.841282 27411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:44.841648 27411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:44.842638 27411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:44.844246 27411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:44.844413 27411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.844573 27411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:44.844653 27411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:44.881541 27411 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:41443
I20260925 10:28:44.881628 27539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:41443 every 8 connection(s)
I20260925 10:28:44.882902 27411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:28:44.885025 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27411
I20260925 10:28:44.885229 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:28:44.885857 13281 external_mini_cluster.cc:1428] Setting key 5160fa719118c815204d41c675588eed
Sep 25 10:28:44 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:44.908854 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:44.886015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:38097 (local address 127.12.248.126:46803)
0925 10:28:44.886517 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:44.886523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:44.887610 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 10:28:44.890508 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:44.890512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:44.890592 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:44.890690 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:44.892994 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.893832 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.896439 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.896899 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.897006 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:44.901072 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:44.901082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:44.901090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:44.901143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:44.905054 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:44.906013 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:44.906021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:44.906035 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:44.906196 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:44.906721 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:44.906729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:44.906735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:44.907143 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:44.907303 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:44.908062 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 10:28:44.908327 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":230,"threads_started":1}
I20260925 10:28:44.909714 27543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:44.886664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:38097)
0925 10:28:44.887450 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:44.887489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:44.890155 (+ 2666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:44.891027 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:44.891042 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:44.891878 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:44.892763 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:44.892792 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.894001 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.894009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:44.896272 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:44.896292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:44.897165 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:44.897173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:44.897410 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:44.898216 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:44.898244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:44.900770 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:44.905230 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:44.905238 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:44.905254 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:44.905871 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:44.906355 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:44.906359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:44.906361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:44.906581 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:44.907284 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:44.907290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:44.907658 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 10:28:44.907899 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 10:28:44.908333 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":351,"threads_started":1}
I20260925 10:28:44.911298 27540 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:44.911612 27540 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:44.912364 27540 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:28:44.914011 27214 ts_manager.cc:194] Registered new tserver with Master: f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
I20260925 10:28:44.914882 27214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:38097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:28:44.945160 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:45.282790 27546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:45.283087 27546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:45.283269 27546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:45.290508 27546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:45.290623 27546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:45.290853 27546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:28:45.300812 27546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:45.302372 27546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:45.303725 27546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:45.317514 27554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:45.317534 27552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:45.317688 27551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:45.318145 27546 server_base.cc:1034] running on GCE node
I20260925 10:28:45.319031 27546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:45.320039 27546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:45.321259 27546 hybrid_clock.cc:648] HybridClock initialized: now 1790332125321206 us; error 42 us; skew 500 ppm
I20260925 10:28:45.321641 27546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:45 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:45.325181 27546 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:28:45.326881 27546 webserver.cc:460] Webserver started at http://127.12.248.67:40017/ using document root <none> and password file <none>
I20260925 10:28:45.327589 27546 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:45.327739 27546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:45.328056 27546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:28:45.331053 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "0134e7e11e1c40c79165d6b72883f70b"
format_stamp: "Formatted at 2026-09-25 10:28:45 on dist-test-slave-f9rl"
server_key: "74f0c84d16459b252dfa85a56dab12fd"
server_key_iv: "cf98e177628605eee10e1892ddd56cab"
server_key_version: "encryptionkey@0"
I20260925 10:28:45.331837 27546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "0134e7e11e1c40c79165d6b72883f70b"
format_stamp: "Formatted at 2026-09-25 10:28:45 on dist-test-slave-f9rl"
server_key: "74f0c84d16459b252dfa85a56dab12fd"
server_key_iv: "cf98e177628605eee10e1892ddd56cab"
server_key_version: "encryptionkey@0"
I20260925 10:28:45.337110 27546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:28:45.340677 27561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:45.342280 27546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:45.342463 27546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0134e7e11e1c40c79165d6b72883f70b"
format_stamp: "Formatted at 2026-09-25 10:28:45 on dist-test-slave-f9rl"
server_key: "74f0c84d16459b252dfa85a56dab12fd"
server_key_iv: "cf98e177628605eee10e1892ddd56cab"
server_key_version: "encryptionkey@0"
I20260925 10:28:45.342674 27546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:45.389578 27405 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
I20260925 10:28:45.393162 27546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:45.398109 27546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:45.398360 27546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:45.399288 27546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:45.400602 27546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:45.400676 27546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:45.400765 27546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:45.400810 27546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:45.435075 27546 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:38501
I20260925 10:28:45.435165 27674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:38501 every 8 connection(s)
I20260925 10:28:45.436386 27546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:28:45.438799 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27546
I20260925 10:28:45.438983 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:28:45.439616 13281 external_mini_cluster.cc:1428] Setting key 5edae2673c6fb10f07d0af8f478138d7
Sep 25 10:28:45 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:45.457788 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.438949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:58741 (local address 127.12.248.126:46803)
0925 10:28:45.439494 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.439508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.440656 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 10:28:45.442563 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.442568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.442653 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.442782 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:45.444308 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.445110 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.446862 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.447370 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.447503 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.451235 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:45.451240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:45.451244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:45.451277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:45.454746 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.455640 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.455644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.455646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.455787 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.456192 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.456195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.456197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.456476 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:45.456598 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.457279 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.457508 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":249,"threads_started":1}
I20260925 10:28:45.458806 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.439654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:58741)
0925 10:28:45.440485 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.440546 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.442296 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.443021 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.443029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.443530 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:45.444141 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.444159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.445273 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.445276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.446704 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.446716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.447612 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.447620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.447856 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.448576 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:45.448601 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:45.451024 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:45.454882 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:45.454891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:45.454909 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:45.455486 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:45.455885 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:45.455889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:45.455891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:45.456089 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:45.456594 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:45.456599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:45.456929 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.457185 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.457649 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":333,"threads_started":1}
I20260925 10:28:45.460290 27675 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:45.460620 27675 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:45.461485 27675 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
I20260925 10:28:45.463034 27214 ts_manager.cc:194] Registered new tserver with Master: 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501)
I20260925 10:28:45.463871 27214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:58741
I20260925 10:28:45.467764 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:28:45.484771 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.472434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.12.248.126:46803)
0925 10:28:45.472672 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.472679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.472998 (+ 319us) server_negotiation.cc:407] Connection header received
0925 10:28:45.473134 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.473138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.473201 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.473280 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:45.474690 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.475398 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.476816 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.477215 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.477306 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.479862 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:45.479867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:45.479870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:45.479901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:45.482390 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.483042 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.483045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.483047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.483160 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.483543 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.483546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.483548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.483855 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:45.484040 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.484285 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.484446 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:28:45.488991 27214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:28:45.496065 27214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:28:45.521370 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.508613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.12.248.66:41443)
0925 10:28:45.509198 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.509208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.509939 (+ 731us) server_negotiation.cc:407] Connection header received
0925 10:28:45.510240 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.510249 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.510533 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.510717 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:45.511946 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.514024 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.517969 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.519545 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.519683 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.519975 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.520814 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.521035 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":287,"threads_started":1}
I20260925 10:28:45.521714 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.508889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38501 (local address 127.0.0.1:60270)
0925 10:28:45.509800 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.509836 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.509996 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.514112 (+ 4116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.514120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.514155 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:45.515299 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.515317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.517008 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.517011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.518923 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.518940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.520682 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.520688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.520875 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.520909 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.521000 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.521197 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":541,"thread_start_us":974,"threads_started":1}
I20260925 10:28:45.521967 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.509574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.12.248.67:38501)
0925 10:28:45.512946 (+ 3372us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.512958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.513002 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:28:45.513124 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.513135 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.513425 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.513699 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:45.515531 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.516850 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.519064 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.520474 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.520615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.520846 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.521064 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.521220 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3037,"thread_start_us":857,"threads_started":1}
I20260925 10:28:45.522358 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.508423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41443 (local address 127.0.0.1:45844)
0925 10:28:45.509794 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.509839 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.510045 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.511102 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.511112 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.511153 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:45.511694 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.511710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.514282 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.514295 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.517774 (+ 3479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.517794 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.520192 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.520200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.520513 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.520592 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.521741 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.521894 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":53,"thread_start_us":1087,"threads_started":1}
I20260925 10:28:45.526543 27692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.510322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59474)
0925 10:28:45.517925 (+ 7603us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.517945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.518090 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.518745 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.518751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.518778 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:45.519229 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.519245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.521489 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.521494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.523629 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.523642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.525801 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.525806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.525923 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.525945 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.526069 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.526202 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7466,"thread_start_us":7309,"threads_started":1}
I20260925 10:28:45.526614 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.510732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.12.248.65:36175)
0925 10:28:45.514940 (+ 4208us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.514947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.518077 (+ 3130us) server_negotiation.cc:407] Connection header received
0925 10:28:45.518250 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.518258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.518544 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.518742 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:45.519649 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.521266 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.523784 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.525307 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.525472 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.525681 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.526117 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.526281 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4061,"thread_start_us":3900,"threads_started":1}
I20260925 10:28:45.526659 27609 tablet_service.cc:1467] Processing CreateTablet for tablet facdd916f02e4dc1973e9b0290c5f03a (DEFAULT_TABLE table=test-table [id=036a67faba8e440c8f6431b9d7b613eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:45.526659 27474 tablet_service.cc:1467] Processing CreateTablet for tablet facdd916f02e4dc1973e9b0290c5f03a (DEFAULT_TABLE table=test-table [id=036a67faba8e440c8f6431b9d7b613eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:45.528232 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet facdd916f02e4dc1973e9b0290c5f03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:45.528229 27609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet facdd916f02e4dc1973e9b0290c5f03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:45.530171 27339 tablet_service.cc:1467] Processing CreateTablet for tablet facdd916f02e4dc1973e9b0290c5f03a (DEFAULT_TABLE table=test-table [id=036a67faba8e440c8f6431b9d7b613eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:45.531538 27339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet facdd916f02e4dc1973e9b0290c5f03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:45.543892 27694 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Bootstrap starting.
I20260925 10:28:45.546329 27696 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Bootstrap starting.
I20260925 10:28:45.547672 27695 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Bootstrap starting.
I20260925 10:28:45.548751 27694 tablet_bootstrap.cc:654] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:45.550271 27696 tablet_bootstrap.cc:654] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:45.551110 27694 log.cc:824] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:45.551199 27695 tablet_bootstrap.cc:654] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:45.551576 27696 log.cc:824] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:45.552523 27695 log.cc:824] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:45.554486 27694 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: No bootstrap required, opened a new log
I20260925 10:28:45.554787 27694 ts_tablet_manager.cc:1397] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 10:28:45.554826 27696 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: No bootstrap required, opened a new log
I20260925 10:28:45.555086 27696 ts_tablet_manager.cc:1397] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:28:45.555544 27695 tablet_bootstrap.cc:492] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: No bootstrap required, opened a new log
I20260925 10:28:45.555811 27695 ts_tablet_manager.cc:1397] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 10:28:45.563112 27694 raft_consensus.cc:348] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.563519 27694 raft_consensus.cc:374] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:45.563633 27694 raft_consensus.cc:729] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0134e7e11e1c40c79165d6b72883f70b, State: Initialized, Role: FOLLOWER
I20260925 10:28:45.564275 27694 consensus_queue.cc:260] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.565423 27675 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
I20260925 10:28:45.566202 27694 ts_tablet_manager.cc:1428] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 10:28:45.566217 27696 raft_consensus.cc:348] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.566687 27696 raft_consensus.cc:374] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:45.566468 27695 raft_consensus.cc:348] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.566777 27696 raft_consensus.cc:729] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a89e2294427da7ebfd14dc257ec4, State: Initialized, Role: FOLLOWER
I20260925 10:28:45.567001 27695 raft_consensus.cc:374] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:45.567123 27695 raft_consensus.cc:729] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c7bf75095482a86d7ed6db4f3bd01, State: Initialized, Role: FOLLOWER
I20260925 10:28:45.567523 27696 consensus_queue.cc:260] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.567778 27695 consensus_queue.cc:260] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.569661 27696 ts_tablet_manager.cc:1428] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260925 10:28:45.570681 27540 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
I20260925 10:28:45.571334 27695 ts_tablet_manager.cc:1428] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260925 10:28:45.608050 27406 tablet.cc:2369] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:28:45.635298 27541 tablet.cc:2369] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:28:45.668358 27702 raft_consensus.cc:482] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:45.668747 27702 raft_consensus.cc:504] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.670739 27702 leader_election.cc:288] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175)
I20260925 10:28:45.677376 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.671158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:37779 (local address 127.12.248.67:38501)
0925 10:28:45.671345 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.671350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.671968 (+ 618us) server_negotiation.cc:407] Connection header received
0925 10:28:45.672252 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.672257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.672341 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.672452 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:45.673110 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.673918 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.675664 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.676430 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.676616 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.676697 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.676979 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.677121 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:28:45.677510 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.671122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38501 (local address 127.12.248.66:37779)
0925 10:28:45.671851 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.671873 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.672026 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.672552 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.672557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.672590 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:45.672953 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.672962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.674063 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.674071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.675514 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.675526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.676659 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.676665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.676819 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.676842 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.676952 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.677086 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":406,"threads_started":1}
I20260925 10:28:45.677932 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.672207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:38877 (local address 127.12.248.65:36175)
0925 10:28:45.672418 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.672425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.672779 (+ 354us) server_negotiation.cc:407] Connection header received
0925 10:28:45.672989 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.672993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.673052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.673150 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:45.673757 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.674485 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.676152 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.676926 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.677114 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.677209 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.677545 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.677673 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:28:45.677958 27704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.672022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.12.248.66:38877)
0925 10:28:45.672672 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:45.672689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:45.672841 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:45.673163 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:45.673168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:45.673191 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:45.673596 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.673606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.674625 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.674634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.675988 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:45.676000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.677135 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.677142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:45.677332 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.677354 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:28:45.677517 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 10:28:45.677644 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":352,"threads_started":1}
I20260925 10:28:45.678915 27629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "facdd916f02e4dc1973e9b0290c5f03a" candidate_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0134e7e11e1c40c79165d6b72883f70b" is_pre_election: true
I20260925 10:28:45.679201 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "facdd916f02e4dc1973e9b0290c5f03a" candidate_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a89e2294427da7ebfd14dc257ec4" is_pre_election: true
I20260925 10:28:45.679499 27629 raft_consensus.cc:2454] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c7bf75095482a86d7ed6db4f3bd01 in term 0.
I20260925 10:28:45.679623 27359 raft_consensus.cc:2454] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74c7bf75095482a86d7ed6db4f3bd01 in term 0.
I20260925 10:28:45.680246 27429 leader_election.cc:302] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4, f74c7bf75095482a86d7ed6db4f3bd01; no voters:
I20260925 10:28:45.680614 27702 raft_consensus.cc:2781] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:28:45.680768 27702 raft_consensus.cc:482] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:28:45.680873 27702 raft_consensus.cc:3037] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:45.683300 27702 raft_consensus.cc:504] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.683893 27702 leader_election.cc:288] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [CANDIDATE]: Term 1 election: Requested vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175)
I20260925 10:28:45.684427 27359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "facdd916f02e4dc1973e9b0290c5f03a" candidate_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f32a89e2294427da7ebfd14dc257ec4"
I20260925 10:28:45.684422 27629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "facdd916f02e4dc1973e9b0290c5f03a" candidate_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0134e7e11e1c40c79165d6b72883f70b"
I20260925 10:28:45.684649 27629 raft_consensus.cc:3037] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:45.684650 27359 raft_consensus.cc:3037] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:45.687932 27629 raft_consensus.cc:2454] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c7bf75095482a86d7ed6db4f3bd01 in term 1.
I20260925 10:28:45.687924 27359 raft_consensus.cc:2454] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74c7bf75095482a86d7ed6db4f3bd01 in term 1.
W20260925 10:28:45.688019 27676 tablet.cc:2369] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:28:45.688356 27429 leader_election.cc:302] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4, f74c7bf75095482a86d7ed6db4f3bd01; no voters:
I20260925 10:28:45.688664 27702 raft_consensus.cc:2781] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:45.689142 27702 raft_consensus.cc:686] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 LEADER]: Becoming Leader. State: Replica: f74c7bf75095482a86d7ed6db4f3bd01, State: Running, Role: LEADER
I20260925 10:28:45.689743 27702 consensus_queue.cc:237] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } }
I20260925 10:28:45.694849 27213 catalog_manager.cc:5495] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } health_report { overall_health: UNKNOWN } } }
I20260925 10:28:45.715412 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.709216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.12.248.66:41443)
0925 10:28:45.709424 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.709452 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.709473 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:28:45.709605 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.709610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.709704 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.709856 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:28:45.710171 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.711327 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.712903 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.713576 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.713721 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.714208 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:28:45.714851 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:28:45.714994 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.715052 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.715132 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:28:45.724097 27359 raft_consensus.cc:1261] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Refusing update from remote peer f74c7bf75095482a86d7ed6db4f3bd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:45.724097 27629 raft_consensus.cc:1261] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Refusing update from remote peer f74c7bf75095482a86d7ed6db4f3bd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:45.725085 27702 consensus_queue.cc:1035] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:45.725418 27705 consensus_queue.cc:1035] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:45.737064 27708 mvcc.cc:205] Tried to move back new op lower bound from 7333200386954649600 to 7333200386828926976. Current Snapshot: MvccSnapshot[applied={T|T < 7333200386954649600}]
I20260925 10:28:45.737751 27709 mvcc.cc:205] Tried to move back new op lower bound from 7333200386954649600 to 7333200386828926976. Current Snapshot: MvccSnapshot[applied={T|T < 7333200386954649600}]
I20260925 10:28:45.751377 27213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:45.751652 27213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:45.755312 27213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Sending DeleteTablet for 3 replicas of tablet facdd916f02e4dc1973e9b0290c5f03a
I20260925 10:28:45.756500 27474 tablet_service.cc:1514] Processing DeleteTablet for tablet facdd916f02e4dc1973e9b0290c5f03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:45 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:45844
I20260925 10:28:45.756618 27609 tablet_service.cc:1514] Processing DeleteTablet for tablet facdd916f02e4dc1973e9b0290c5f03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:45 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:60270
I20260925 10:28:45.756803 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27275
I20260925 10:28:45.757597 27339 tablet_service.cc:1514] Processing DeleteTablet for tablet facdd916f02e4dc1973e9b0290c5f03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:45 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59474
I20260925 10:28:45.757893 27717 tablet_replica.cc:330] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: stopping tablet replica
I20260925 10:28:45.758316 27717 raft_consensus.cc:2229] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:28:45.759008 27717 raft_consensus.cc:2258] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:45.759800 27718 tablet_replica.cc:330] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: stopping tablet replica
I20260925 10:28:45.760246 27718 raft_consensus.cc:2229] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:45.760772 27718 raft_consensus.cc:2258] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:45.763060 27718 ts_tablet_manager.cc:1905] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:45.763060 27717 ts_tablet_manager.cc:1905] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:45.770864 27720 tablet_replica.cc:330] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: stopping tablet replica
I20260925 10:28:45.771371 27720 raft_consensus.cc:2229] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:45.771540 27717 ts_tablet_manager.cc:1918] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:45.771741 27717 log.cc:1196] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/facdd916f02e4dc1973e9b0290c5f03a
I20260925 10:28:45.771974 27720 raft_consensus.cc:2258] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:45.772358 27717 ts_tablet_manager.cc:1939] T facdd916f02e4dc1973e9b0290c5f03a P f74c7bf75095482a86d7ed6db4f3bd01: Deleting consensus metadata
I20260925 10:28:45.774541 27201 catalog_manager.cc:4841] TS f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): tablet facdd916f02e4dc1973e9b0290c5f03a (table test-table [id=036a67faba8e440c8f6431b9d7b613eb]) successfully deleted
I20260925 10:28:45.777122 27718 ts_tablet_manager.cc:1918] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:45.777371 27718 log.cc:1196] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/facdd916f02e4dc1973e9b0290c5f03a
I20260925 10:28:45.778013 27718 ts_tablet_manager.cc:1939] T facdd916f02e4dc1973e9b0290c5f03a P 0134e7e11e1c40c79165d6b72883f70b: Deleting consensus metadata
I20260925 10:28:45.778345 27720 ts_tablet_manager.cc:1905] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:45.779891 27200 catalog_manager.cc:4841] TS 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): tablet facdd916f02e4dc1973e9b0290c5f03a (table test-table [id=036a67faba8e440c8f6431b9d7b613eb]) successfully deleted
Sep 25 10:28:45 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:28:45.784659 27720 ts_tablet_manager.cc:1918] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:45.784861 27720 log.cc:1196] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/facdd916f02e4dc1973e9b0290c5f03a
I20260925 10:28:45.785392 27720 ts_tablet_manager.cc:1939] T facdd916f02e4dc1973e9b0290c5f03a P 2f32a89e2294427da7ebfd14dc257ec4: Deleting consensus metadata
I20260925 10:28:45.787027 27200 catalog_manager.cc:4841] TS 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175): tablet facdd916f02e4dc1973e9b0290c5f03a (table test-table [id=036a67faba8e440c8f6431b9d7b613eb]) successfully deleted
I20260925 10:28:45.790176 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.758069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.12.248.65:36175)
0925 10:28:45.758286 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.758295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.770153 (+ 11858us) server_negotiation.cc:407] Connection header received
0925 10:28:45.770365 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.770373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.770494 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.770632 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:45.773055 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.774222 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.776234 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.776972 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.777112 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.782672 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:45.782687 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:45.782708 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:45.782759 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:45.787120 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.788049 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.788056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.788059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.788197 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.788610 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.788614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.788615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.789028 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:45.789341 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.789689 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.789884 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:28:45.791288 27399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:45.885489 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27275
I20260925 10:28:45.911247 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27411
Sep 25 10:28:45 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:28:45.934489 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:45.912134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.12.248.66:41443)
0925 10:28:45.912322 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:45.912327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:45.912368 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:28:45.912576 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:45.912580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:45.912697 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:45.912785 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:45.914507 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.915381 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.917264 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:45.917801 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:45.917899 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:45.928194 (+ 10295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:45.928211 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:45.928228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:45.928270 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:45.931540 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.932269 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.932278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.932280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.932430 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:45.932832 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:45.932838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:45.932841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:45.933188 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:45.933586 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:45.933876 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:28:45.934106 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:28:45.935277 27535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:46.050267 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27411
I20260925 10:28:46.069304 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27546
Sep 25 10:28:46 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:28:46.084096 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:46.070093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.12.248.67:38501)
0925 10:28:46.070286 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:46.070290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:46.070309 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:28:46.070452 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:46.070455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:46.070542 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:46.070629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:46.071893 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:46.072613 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:46.073886 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:46.074254 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:46.074349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:46.078637 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:46.078651 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:46.078668 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:46.078699 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:46.081696 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:46.082296 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:46.082302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:46.082304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:46.082430 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:46.082740 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:46.082743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:46.082745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:46.083085 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:46.083306 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:46.083560 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:28:46.083717 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:28:46.084750 27670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:46.176484 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27546
I20260925 10:28:46.193492 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27182
I20260925 10:28:46.194656 27244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:46.307102 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27182
I20260925 10:28:46.328624 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--webserver_interface=127.12.248.126
--webserver_port=42387
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:46.658742 27725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:46.659081 27725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:46.659246 27725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:46.666723 27725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:46.666875 27725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:46.666996 27725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:46.667094 27725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:46.667205 27725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:46.677315 27725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:46803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=42387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:46.678689 27725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:46.680053 27725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:46.703536 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:46.703606 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:46.703552 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:46.704236 27725 server_base.cc:1034] running on GCE node
I20260925 10:28:46.705119 27725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:46.706727 27725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:46.707932 27725 hybrid_clock.cc:648] HybridClock initialized: now 1790332126707857 us; error 70 us; skew 500 ppm
I20260925 10:28:46.708356 27725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:46 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332126, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:46.711942 27725 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:46.713603 27725 webserver.cc:460] Webserver started at http://127.12.248.126:42387/ using document root <none> and password file <none>
I20260925 10:28:46.714383 27725 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:46.714514 27725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:46.720394 27725 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 10:28:46.723416 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:46.724838 27725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:28:46.725021 27725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "65ae6424aad74bf9a68f489b6f26f399"
format_stamp: "Formatted at 2026-09-25 10:28:43 on dist-test-slave-f9rl"
server_key: "46fe916f70e21327ce4bad8d425d9d5e"
server_key_iv: "bc464c0c0666bde366efdf3c534e3b0d"
server_key_version: "encryptionkey@0"
I20260925 10:28:46.726174 27725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:46.745996 27725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:46.750149 27725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:46.750450 27725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:46.769994 27725 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:46803
I20260925 10:28:46.770036 27792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:46803 every 8 connection(s)
I20260925 10:28:46.771644 27725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:46.778774 27793 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:28:46.782014 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27725
I20260925 10:28:46.782193 27793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap starting.
I20260925 10:28:46.782879 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:36175
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=41941
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:46.802315 27793 log.cc:824] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:46.814636 27793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:28:46.815188 27793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap complete.
I20260925 10:28:46.824541 27793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:46.825495 27793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Initialized, Role: FOLLOWER
I20260925 10:28:46.826190 27793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:46.826468 27793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:46.826619 27793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:46.826799 27793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:28:46.831055 27793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:46.831666 27793 leader_election.cc:302] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ae6424aad74bf9a68f489b6f26f399; no voters:
I20260925 10:28:46.832309 27793 leader_election.cc:288] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:28:46.832563 27797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:28:46.833761 27797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 LEADER]: Becoming Leader. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Running, Role: LEADER
I20260925 10:28:46.834613 27793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:46.834540 27797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:46.837778 27798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:46.837754 27799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ae6424aad74bf9a68f489b6f26f399. Latest consensus state: current_term: 2 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:46.838207 27798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:46.838248 27799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:46.852643 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:46.866526 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:46.868647 27803 catalog_manager.cc:1259] Loaded cluster ID: 15b0e781d01c41eaaaad02b3a66a07cc
I20260925 10:28:46.869201 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:28:46.873790 27803 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:28:46.876061 27814 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:28:46.886550 27803 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:28:46.894120 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:28:46.895537 27803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Loaded TSK: 0
I20260925 10:28:46.897125 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:28:47.170457 27795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:47.170753 27795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:47.170929 27795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:47.179126 27795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:47.179248 27795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:47.179446 27795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:28:47.190233 27795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:36175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=41941
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:47.191586 27795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:47.192968 27795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:47.208098 27821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:47.208259 27820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:47.208077 27823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:47.209028 27795 server_base.cc:1034] running on GCE node
I20260925 10:28:47.209839 27795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:47.211087 27795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:47.212354 27795 hybrid_clock.cc:648] HybridClock initialized: now 1790332127212302 us; error 36 us; skew 500 ppm
I20260925 10:28:47.212751 27795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:47 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:47.222319 27795 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:28:47.224085 27795 webserver.cc:460] Webserver started at http://127.12.248.65:41941/ using document root <none> and password file <none>
I20260925 10:28:47.224823 27795 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:47.224979 27795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:47.230880 27795 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:28:47.233731 27830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.235241 27795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:28:47.235419 27795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "2f32a89e2294427da7ebfd14dc257ec4"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "a4780896b7d1dece83782a8d7bd58df1"
server_key_iv: "5520e3d71781e1c326d88dc60b9ae53c"
server_key_version: "encryptionkey@0"
I20260925 10:28:47.236538 27795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:47.255252 27795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:47.259737 27795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:47.260035 27795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:47.261067 27795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:47.262671 27795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:47.262799 27795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.262938 27795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:47.263028 27795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.296890 27795 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36175
I20260925 10:28:47.297012 27944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36175 every 8 connection(s)
I20260925 10:28:47.298656 27795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:28:47.307674 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27795
I20260925 10:28:47.308400 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:41443
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=37761
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:28:47 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:47.323239 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:47.301757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:38167)
0925 10:28:47.302582 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:47.302634 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:47.304413 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:47.305759 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:47.305766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:47.306296 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:47.306829 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:47.306846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.309273 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.309277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:47.310552 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:47.310565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.312003 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.312008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:47.312218 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:47.312928 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:47.312951 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:47.315282 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:47.318735 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:47.318744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:47.318759 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:47.319206 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:47.319659 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:47.319666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:47.319670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:47.319859 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:47.320608 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:47.320616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:47.320955 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 10:28:47.321235 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 10:28:47.321885 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":359,"threads_started":1}
I20260925 10:28:47.323239 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:47.301564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:38167 (local address 127.12.248.126:46803)
0925 10:28:47.302191 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:47.302202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:47.302796 (+ 594us) server_negotiation.cc:407] Connection header received
0925 10:28:47.304910 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:47.304923 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:47.305264 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:47.305618 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:47.307061 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.309085 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.310698 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.311789 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.312006 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:47.315585 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:47.315593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:47.315609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:47.315646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:47.318580 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:47.319362 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:47.319369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:47.319374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:47.319540 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:47.319988 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:47.319993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:47.319995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:47.320461 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:47.320672 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:47.321492 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 10:28:47.321924 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":285,"threads_started":1}
I20260925 10:28:47.326503 27945 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:47.326848 27945 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:47.327627 27945 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
I20260925 10:28:47.330175 27758 ts_manager.cc:194] Registered new tserver with Master: 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175)
I20260925 10:28:47.331804 27758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:38167
W20260925 10:28:47.657275 27949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:47.657661 27949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:47.657820 27949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:47.665115 27949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:47.665278 27949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:47.665524 27949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:28:47.675762 27949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:41443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=37761
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:47.677229 27949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:47.678632 27949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:47.691855 27954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:47.691900 27955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:47.691871 27957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:47.692716 27949 server_base.cc:1034] running on GCE node
I20260925 10:28:47.693324 27949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:47.694289 27949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:47.695528 27949 hybrid_clock.cc:648] HybridClock initialized: now 1790332127695477 us; error 41 us; skew 500 ppm
I20260925 10:28:47.695873 27949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:47 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:47.699173 27949 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:28:47.700677 27949 webserver.cc:460] Webserver started at http://127.12.248.66:37761/ using document root <none> and password file <none>
I20260925 10:28:47.701380 27949 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:47.701560 27949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:47.707059 27949 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 10:28:47.709846 27964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.711238 27949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:28:47.711428 27949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "7b4ad05bbb32e23f0a676bec5f72a4c7"
server_key_iv: "401389b645ab97ee23d2cd6fe8156934"
server_key_version: "encryptionkey@0"
I20260925 10:28:47.712414 27949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:47.734542 27949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:47.739054 27949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:47.739363 27949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:47.740512 27949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:47.742054 27949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:47.742173 27949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.742308 27949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:47.742403 27949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:47.777426 27949 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:41443
I20260925 10:28:47.777518 28077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:41443 every 8 connection(s)
I20260925 10:28:47.778878 27949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:28:47.781831 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 27949
I20260925 10:28:47.782560 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:38501
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=40017
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:28:47 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:47.804224 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:47.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:59563 (local address 127.12.248.126:46803)
0925 10:28:47.785684 (+ 3741us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:47.785693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:47.785719 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:28:47.786645 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:47.786654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:47.786779 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:47.786905 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:47.789228 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.790470 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.793000 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.793421 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.793589 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:47.797999 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:47.798011 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:47.798014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:47.798048 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:47.800711 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:47.801698 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:47.801706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:47.801711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:47.801881 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:47.802437 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:47.802441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:47.802443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:47.802722 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:47.802880 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:47.803613 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 10:28:47.803861 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3570}
I20260925 10:28:47.805234 28080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:47.782538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:59563)
0925 10:28:47.783475 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:47.783534 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:47.786285 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:47.787225 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:47.787238 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:47.788143 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:47.788985 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:47.789014 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.790658 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.790664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:47.792831 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:47.792863 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:47.793731 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:47.793742 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:47.794117 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:47.794893 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:47.794937 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:47.797770 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:47.800855 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:47.800865 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:47.800883 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:47.801578 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:47.802000 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:47.802005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:47.802008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:47.802296 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:47.802856 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:47.802863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:47.803233 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 10:28:47.803517 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 10:28:47.804097 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260925 10:28:47.806860 28078 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:47.807255 28078 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:47.808382 28078 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
I20260925 10:28:47.810345 27758 ts_manager.cc:194] Registered new tserver with Master: f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
I20260925 10:28:47.811251 27758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:59563
W20260925 10:28:48.128576 28081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:48.128895 28081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:48.129065 28081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:48.137221 28081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:48.137341 28081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:48.137552 28081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:28:48.148499 28081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:38501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=40017
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:48.149961 28081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:48.151374 28081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:48.165096 28086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:48.165081 28087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:48.165221 28089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:48.166393 28081 server_base.cc:1034] running on GCE node
I20260925 10:28:48.167049 28081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:48.168048 28081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:48.169291 28081 hybrid_clock.cc:648] HybridClock initialized: now 1790332128169250 us; error 53 us; skew 500 ppm
I20260925 10:28:48.169809 28081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:48 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332128, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:48.173122 28081 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:28:48.174757 28081 webserver.cc:460] Webserver started at http://127.12.248.67:40017/ using document root <none> and password file <none>
I20260925 10:28:48.175467 28081 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:48.175629 28081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:48.181118 28081 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 10:28:48.184119 28096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:48.185765 28081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:48.185994 28081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0134e7e11e1c40c79165d6b72883f70b"
format_stamp: "Formatted at 2026-09-25 10:28:45 on dist-test-slave-f9rl"
server_key: "74f0c84d16459b252dfa85a56dab12fd"
server_key_iv: "cf98e177628605eee10e1892ddd56cab"
server_key_version: "encryptionkey@0"
I20260925 10:28:48.187144 28081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:48.207643 28081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:48.212186 28081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:48.212487 28081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:48.213636 28081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:48.215057 28081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:48.215194 28081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:48.215341 28081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:48.215440 28081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:48.250036 28081 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:38501
I20260925 10:28:48.250087 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:38501 every 8 connection(s)
I20260925 10:28:48.251603 28081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:28:48.261185 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28081
Sep 25 10:28:48 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332128, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:28:48.274283 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.254376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:45141 (local address 127.12.248.126:46803)
0925 10:28:48.254557 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.254561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.255866 (+ 1305us) server_negotiation.cc:407] Connection header received
0925 10:28:48.257757 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.257762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.257852 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.257962 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:48.259455 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.260189 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.262309 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.263108 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.263245 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.267209 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:48.267215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:48.267220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:48.267258 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:48.270814 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.271532 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.271538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.271543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.271703 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.272170 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.272174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.272176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.272462 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:48.272578 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.273532 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.273756 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:28:48.275514 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.254921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:45141)
0925 10:28:48.255718 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.255755 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.257481 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.258203 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.258211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.258733 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:48.259285 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.259302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.260354 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.260364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.262149 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.262164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.263322 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.263326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.263692 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.264412 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:48.264446 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:48.266994 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:48.270982 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:48.270988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:48.270999 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:48.271398 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:48.271813 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:48.271816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:48.271819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:48.272049 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:48.272609 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:48.272620 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:48.273055 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.273373 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.274029 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":307,"threads_started":1}
I20260925 10:28:48.277133 28210 heartbeater.cc:347] Connected to a master server at 127.12.248.126:46803
I20260925 10:28:48.277490 28210 heartbeater.cc:464] Registering TS with master...
I20260925 10:28:48.278379 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.262227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.12.248.126:46803)
0925 10:28:48.263125 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.263130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.263150 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:28:48.263199 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.263207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.263275 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.263385 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:48.265492 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.266272 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.268027 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.268603 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.268700 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.272062 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:48.272069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:48.272075 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:48.272118 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:48.275339 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.276194 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.276199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.276202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.276352 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.276768 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.276772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.276774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.277078 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:48.277331 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.277733 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.277991 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":616,"threads_started":1}
I20260925 10:28:48.278436 28210 heartbeater.cc:511] Master 127.12.248.126:46803 requested a full tablet report, sending...
I20260925 10:28:48.279853 27758 ts_manager.cc:194] Registered new tserver with Master: 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501)
I20260925 10:28:48.280750 27758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:45141
I20260925 10:28:48.281096 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:28:48.287657 28214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.282043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.12.248.126:46803)
0925 10:28:48.282227 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.282234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.282700 (+ 466us) server_negotiation.cc:407] Connection header received
0925 10:28:48.282853 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.282858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.283027 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.283179 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:28:48.283553 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.284211 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.285610 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.286055 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.286152 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.286673 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:28:48.287205 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:28:48.287310 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.287398 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.287460 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:28:48.288553 27758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:28:48.296130 27758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:28:48.319540 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.308419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.12.248.67:38501)
0925 10:28:48.310622 (+ 2203us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.310635 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.310674 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:28:48.310794 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.310806 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.311107 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.311343 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:48.312188 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.313912 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.316823 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.318289 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.318420 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.318668 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.319003 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.319170 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1969,"thread_start_us":1783,"threads_started":1}
I20260925 10:28:48.319530 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.308067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38501 (local address 127.0.0.1:60286)
0925 10:28:48.308901 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.308942 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.309107 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.311393 (+ 2286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.311404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.311448 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:48.311913 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.311926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.314034 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.314042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.316614 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.316628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.318508 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.318515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.318737 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.318805 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.318958 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.319186 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":457,"threads_started":1}
I20260925 10:28:48.321425 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.308696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41443 (local address 127.0.0.1:45864)
0925 10:28:48.309789 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.309812 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.309945 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.314475 (+ 4530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.314479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.314514 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:48.314934 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.314943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.316748 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.316754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.318684 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.318699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.320568 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.320574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.320766 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.320797 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.320860 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.321000 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":44,"thread_start_us":838,"threads_started":1}
I20260925 10:28:48.321987 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.309990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.12.248.65:36175)
0925 10:28:48.312232 (+ 2242us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.312244 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.313067 (+ 823us) server_negotiation.cc:407] Connection header received
0925 10:28:48.313202 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.313213 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.313559 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.313768 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:48.314679 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.316398 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.319150 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.320581 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.320707 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.320929 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.321512 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.321671 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2025,"thread_start_us":1859,"threads_started":1}
I20260925 10:28:48.322068 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.309299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59490)
0925 10:28:48.312747 (+ 3448us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.312770 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.312916 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.313979 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.313987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.314014 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:48.314451 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.314463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.316586 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.316591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.318971 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.318985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.321095 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.321103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.321271 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.321307 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.321415 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.321594 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3293,"mutex_wait_us":3,"thread_start_us":4155,"threads_started":1}
I20260925 10:28:48.322651 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.309232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.12.248.66:41443)
0925 10:28:48.314007 (+ 4775us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.314015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.314042 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:28:48.314136 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.314141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.314289 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.314495 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:48.315118 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.316582 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.318846 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.320310 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.321889 (+ 1579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.322078 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.322159 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.322331 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4604,"thread_start_us":4467,"threads_started":1}
I20260925 10:28:48.324215 28145 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 (DEFAULT_TABLE table=test-table [id=b4c2b13b80ef4af5924056f29f3dfe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:48.325824 28145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88628ac95d4f5e835ddd6afe1e1d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:48.325519 27880 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 (DEFAULT_TABLE table=test-table [id=b4c2b13b80ef4af5924056f29f3dfe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:48.326226 28013 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 (DEFAULT_TABLE table=test-table [id=b4c2b13b80ef4af5924056f29f3dfe1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:28:48.326949 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88628ac95d4f5e835ddd6afe1e1d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:48.327531 28013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88628ac95d4f5e835ddd6afe1e1d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:28:48.334913 27945 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
I20260925 10:28:48.339572 28226 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Bootstrap starting.
I20260925 10:28:48.342918 28227 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Bootstrap starting.
I20260925 10:28:48.343813 28226 tablet_bootstrap.cc:654] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:48.345544 28226 log.cc:824] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:48.346235 28228 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Bootstrap starting.
I20260925 10:28:48.346767 28227 tablet_bootstrap.cc:654] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:48.348351 28227 log.cc:824] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:48.349185 28228 tablet_bootstrap.cc:654] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Neither blocks nor log segments found. Creating new log.
I20260925 10:28:48.350521 28228 log.cc:824] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:48.352174 28226 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: No bootstrap required, opened a new log
I20260925 10:28:48.352191 28227 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: No bootstrap required, opened a new log
I20260925 10:28:48.352470 28227 ts_tablet_manager.cc:1397] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 10:28:48.352514 28226 ts_tablet_manager.cc:1397] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 10:28:48.353605 28228 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: No bootstrap required, opened a new log
I20260925 10:28:48.353871 28228 ts_tablet_manager.cc:1397] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 10:28:48.360033 28227 raft_consensus.cc:348] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.360424 28227 raft_consensus.cc:374] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:48.360548 28227 raft_consensus.cc:729] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c7bf75095482a86d7ed6db4f3bd01, State: Initialized, Role: FOLLOWER
I20260925 10:28:48.361191 28227 consensus_queue.cc:260] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.362479 28078 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
I20260925 10:28:48.363209 28227 ts_tablet_manager.cc:1428] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260925 10:28:48.364573 28228 raft_consensus.cc:348] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.364477 28226 raft_consensus.cc:348] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.364966 28226 raft_consensus.cc:374] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:48.365041 28228 raft_consensus.cc:374] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:28:48.365115 28226 raft_consensus.cc:729] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a89e2294427da7ebfd14dc257ec4, State: Initialized, Role: FOLLOWER
I20260925 10:28:48.365175 28228 raft_consensus.cc:729] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0134e7e11e1c40c79165d6b72883f70b, State: Initialized, Role: FOLLOWER
I20260925 10:28:48.365829 28226 consensus_queue.cc:260] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.365808 28228 consensus_queue.cc:260] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.367781 28228 ts_tablet_manager.cc:1428] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260925 10:28:48.368075 28226 ts_tablet_manager.cc:1428] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260925 10:28:48.368700 28210 heartbeater.cc:503] Master 127.12.248.126:46803 was elected leader, sending a full tablet report...
W20260925 10:28:48.503250 28211 tablet.cc:2369] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:28:48.531163 28079 tablet.cc:2369] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:28:48.551250 27946 tablet.cc:2369] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:28:48.625672 28235 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:48.626051 28235 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.627874 28235 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
I20260925 10:28:48.634652 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.628262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:48937 (local address 127.12.248.67:38501)
0925 10:28:48.628468 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.628475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.629240 (+ 765us) server_negotiation.cc:407] Connection header received
0925 10:28:48.629507 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.629514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.629586 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.629707 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:48.630350 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.631090 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.632866 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.633672 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.633861 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.633935 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.634231 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.634358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:28:48.634754 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.628374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38501 (local address 127.12.248.65:48937)
0925 10:28:48.629129 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.629144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.629326 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.629771 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.629776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.629801 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:48.630199 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.630216 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.631231 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.631240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.632709 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.632721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.633877 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.633883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.634053 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.634077 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.634204 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.634354 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":384,"threads_started":1}
I20260925 10:28:48.634954 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.628376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:40063 (local address 127.12.248.66:41443)
0925 10:28:48.628554 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.628558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.629684 (+ 1126us) server_negotiation.cc:407] Connection header received
0925 10:28:48.629911 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.629915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.629973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.630085 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:28:48.630646 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.631452 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.633135 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.633967 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.634159 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.634232 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.634575 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.634700 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:28:48.635129 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.628380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41443 (local address 127.12.248.65:40063)
0925 10:28:48.629573 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.629587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:48.629751 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:48.630120 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:48.630125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:48.630150 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:28:48.630478 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.630487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.631594 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.631603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.632977 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:48.632990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.634189 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.634195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:48.634383 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.634434 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:28:48.634548 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:28:48.634675 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":122,"thread_start_us":725,"threads_started":1}
I20260925 10:28:48.636051 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88628ac95d4f5e835ddd6afe1e1d62" candidate_uuid: "2f32a89e2294427da7ebfd14dc257ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0134e7e11e1c40c79165d6b72883f70b" is_pre_election: true
I20260925 10:28:48.636147 28033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88628ac95d4f5e835ddd6afe1e1d62" candidate_uuid: "2f32a89e2294427da7ebfd14dc257ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" is_pre_election: true
I20260925 10:28:48.636574 28165 raft_consensus.cc:2454] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f32a89e2294427da7ebfd14dc257ec4 in term 0.
I20260925 10:28:48.636598 28033 raft_consensus.cc:2454] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f32a89e2294427da7ebfd14dc257ec4 in term 0.
I20260925 10:28:48.637228 27833 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0134e7e11e1c40c79165d6b72883f70b, 2f32a89e2294427da7ebfd14dc257ec4; no voters:
I20260925 10:28:48.637691 28235 raft_consensus.cc:2781] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:28:48.637837 28235 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:28:48.637948 28235 raft_consensus.cc:3037] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:48.640193 28235 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.640802 28235 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 1 election: Requested vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
I20260925 10:28:48.641224 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88628ac95d4f5e835ddd6afe1e1d62" candidate_uuid: "2f32a89e2294427da7ebfd14dc257ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0134e7e11e1c40c79165d6b72883f70b"
I20260925 10:28:48.641288 28033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88628ac95d4f5e835ddd6afe1e1d62" candidate_uuid: "2f32a89e2294427da7ebfd14dc257ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
I20260925 10:28:48.641499 28165 raft_consensus.cc:3037] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:48.641549 28033 raft_consensus.cc:3037] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:28:48.643898 28033 raft_consensus.cc:2454] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f32a89e2294427da7ebfd14dc257ec4 in term 1.
I20260925 10:28:48.643898 28165 raft_consensus.cc:2454] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f32a89e2294427da7ebfd14dc257ec4 in term 1.
I20260925 10:28:48.644337 27834 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0134e7e11e1c40c79165d6b72883f70b, 2f32a89e2294427da7ebfd14dc257ec4, f74c7bf75095482a86d7ed6db4f3bd01; no voters:
I20260925 10:28:48.644685 28235 raft_consensus.cc:2781] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:28:48.645206 28235 raft_consensus.cc:686] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f32a89e2294427da7ebfd14dc257ec4, State: Running, Role: LEADER
I20260925 10:28:48.645767 28235 consensus_queue.cc:237] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:48.650311 27758 catalog_manager.cc:5495] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "2f32a89e2294427da7ebfd14dc257ec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } health_report { overall_health: UNKNOWN } } }
I20260925 10:28:48.692775 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.686798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.12.248.65:36175)
0925 10:28:48.687030 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.687037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.687153 (+ 116us) server_negotiation.cc:407] Connection header received
0925 10:28:48.687423 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.687427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.687530 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.687695 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:28:48.688065 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.688832 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.690473 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.690876 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.691014 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.691637 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:28:48.692157 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:28:48.692326 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.692405 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.692490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:28:48.701360 28033 raft_consensus.cc:1261] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Refusing update from remote peer 2f32a89e2294427da7ebfd14dc257ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:48.701360 28165 raft_consensus.cc:1261] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Refusing update from remote peer 2f32a89e2294427da7ebfd14dc257ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:28:48.702514 28238 consensus_queue.cc:1035] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:48.702849 28235 consensus_queue.cc:1035] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:28:48.731915 27757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:48.732266 27757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:28:48.736997 27757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Sending DeleteTablet for 3 replicas of tablet 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:48.738250 28013 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:45864
I20260925 10:28:48.738605 28144 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:60286
I20260925 10:28:48.738767 28145 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:60286
I20260925 10:28:48.738996 27880 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:28:48 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59490
I20260925 10:28:48.740403 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27795
I20260925 10:28:48.749558 28251 tablet_replica.cc:330] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: stopping tablet replica
I20260925 10:28:48.750216 28251 raft_consensus.cc:2229] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:48.750964 28251 raft_consensus.cc:2258] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:48.752170 28252 tablet_replica.cc:330] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: stopping tablet replica
I20260925 10:28:48.752635 28252 raft_consensus.cc:2229] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:28:48.753132 28252 raft_consensus.cc:2258] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:48.755142 28251 ts_tablet_manager.cc:1905] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:48.756227 28252 ts_tablet_manager.cc:1905] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:28:48.762615 28251 ts_tablet_manager.cc:1918] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:48.762862 28251 log.cc:1196] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:48.763507 28251 ts_tablet_manager.cc:1939] T 7d88628ac95d4f5e835ddd6afe1e1d62 P f74c7bf75095482a86d7ed6db4f3bd01: Deleting consensus metadata
I20260925 10:28:48.764081 28252 ts_tablet_manager.cc:1918] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:28:48.764766 28252 log.cc:1196] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:48.765626 28252 ts_tablet_manager.cc:1939] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 0134e7e11e1c40c79165d6b72883f70b: Deleting consensus metadata
I20260925 10:28:48.765583 27745 catalog_manager.cc:4841] TS f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): tablet 7d88628ac95d4f5e835ddd6afe1e1d62 (table test-table [id=b4c2b13b80ef4af5924056f29f3dfe1d]) successfully deleted
I20260925 10:28:48.767365 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.741362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.12.248.65:36175)
0925 10:28:48.741596 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.741605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.741784 (+ 179us) server_negotiation.cc:407] Connection header received
0925 10:28:48.742002 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.742022 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.742123 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.742281 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:48.744637 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.746249 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.748623 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.749603 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.749762 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.753344 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:48.753359 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:48.753377 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:48.753420 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:48.757854 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.758720 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.758729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.758733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.758914 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.764677 (+ 5763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.764683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.764687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.765180 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:48.765530 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.766474 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.766752 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:28:48.767397 27744 catalog_manager.cc:4841] TS 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): tablet 7d88628ac95d4f5e835ddd6afe1e1d62 (table test-table [id=b4c2b13b80ef4af5924056f29f3dfe1d]) successfully deleted
I20260925 10:28:48.767475 28253 tablet_replica.cc:330] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: stopping tablet replica
I20260925 10:28:48.767920 28253 raft_consensus.cc:2229] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:28:48.768162 27940 generic_service.cc:196] Checking for leaks (request via RPC)
W20260925 10:28:48.768793 27744 catalog_manager.cc:4808] TS 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): delete failed for tablet 7d88628ac95d4f5e835ddd6afe1e1d62 because the tablet was not found. No further retry: Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:48.875242 28253 raft_consensus.cc:2258] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:28:48.875718 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27795
W20260925 10:28:48.894157 27744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.248.65:36175 (error 108)
I20260925 10:28:48.894971 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27949
I20260925 10:28:48.896402 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.895552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59520)
0925 10:28:48.895985 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.896195 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":207,"threads_started":1}
W20260925 10:28:48.896687 27744 catalog_manager.cc:4584] TS 2f32a89e2294427da7ebfd14dc257ec4 (127.12.248.65:36175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d88628ac95d4f5e835ddd6afe1e1d62: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
I20260925 10:28:48.908690 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.895778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.12.248.66:41443)
0925 10:28:48.895973 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:48.895980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:48.896099 (+ 119us) server_negotiation.cc:407] Connection header received
0925 10:28:48.896250 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:48.896256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:48.896347 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:48.896470 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:48.897966 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.898659 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.900069 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:48.900463 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:48.900564 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:48.903291 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:48.903305 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:48.903319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:48.903349 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:48.906021 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.906861 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.906868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.906870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.907003 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:48.907325 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:48.907328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:48.907330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:48.907695 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:48.907905 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:48.908154 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 10:28:48.908310 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:28:48.909361 28073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:48.950536 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.950076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59532)
0925 10:28:48.950288 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.950382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:28:48.951614 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:48.951243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59544)
0925 10:28:48.951392 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:48.951483 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 10:28:49.005124 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27949
I20260925 10:28:49.012627 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:49.011656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59552)
0925 10:28:49.012045 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:49.012275 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 10:28:49.014437 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:49.013677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59568)
0925 10:28:49.014139 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:49.014246 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260925 10:28:49.026480 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28081
I20260925 10:28:49.039916 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:49.027276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.12.248.67:38501)
0925 10:28:49.027469 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:49.027473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:49.027492 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:28:49.027608 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:49.027612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:49.027675 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:49.027778 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:49.029090 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:49.029916 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:49.031281 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:49.031647 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:49.031738 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:49.034317 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:49.034330 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:49.034346 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:49.034373 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:49.037278 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:49.038013 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:49.038021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:49.038023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:49.038165 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:49.038514 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:49.038517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:49.038519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:49.038859 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:49.039094 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:49.039354 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 10:28:49.039539 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:28:49.040546 28205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:49.095889 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:49.095314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59574)
0925 10:28:49.095648 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:49.095735 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 10:28:49.096989 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:49.096590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36175 (local address 127.0.0.1:59578)
0925 10:28:49.096793 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:49.096859 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 10:28:49.134618 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28081
I20260925 10:28:49.151787 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 27725
I20260925 10:28:49.153144 27788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:28:49.268461 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 27725
I20260925 10:28:49.290612 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--webserver_interface=127.12.248.126
--webserver_port=42387
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:46803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:28:49.623330 28260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:49.623606 28260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:49.623696 28260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:49.630865 28260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:28:49.631024 28260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:49.631170 28260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:49.631332 28260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:28:49.631407 28260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:28:49.644284 28260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:46803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=42387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:49.645717 28260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:49.647032 28260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:49.659082 28265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:49.659128 28268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:49.659040 28266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:49.659987 28260 server_base.cc:1034] running on GCE node
I20260925 10:28:49.660938 28260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:49.662864 28260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:49.664088 28260 hybrid_clock.cc:648] HybridClock initialized: now 1790332129664028 us; error 50 us; skew 500 ppm
I20260925 10:28:49.664515 28260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:49 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332129, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:49.667685 28260 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:28:49.669191 28260 webserver.cc:460] Webserver started at http://127.12.248.126:42387/ using document root <none> and password file <none>
I20260925 10:28:49.669948 28260 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:49.670094 28260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:49.675467 28260 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:28:49.678289 28275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:49.679608 28260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:49.679795 28260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "65ae6424aad74bf9a68f489b6f26f399"
format_stamp: "Formatted at 2026-09-25 10:28:43 on dist-test-slave-f9rl"
server_key: "46fe916f70e21327ce4bad8d425d9d5e"
server_key_iv: "bc464c0c0666bde366efdf3c534e3b0d"
server_key_version: "encryptionkey@0"
I20260925 10:28:49.680792 28260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:49.719940 28260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:49.724005 28260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:49.724305 28260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:49.743820 28260 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:46803
I20260925 10:28:49.743847 28326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:46803 every 8 connection(s)
I20260925 10:28:49.745304 28260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:28:49.752343 28327 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:28:49.754092 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28260
I20260925 10:28:49.755107 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:36175
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=41941
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:49.756888 28327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap starting.
I20260925 10:28:49.779772 28327 log.cc:824] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:49.796248 28327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:28:49.796828 28327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399: Bootstrap complete.
I20260925 10:28:49.806213 28327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:49.807135 28327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Initialized, Role: FOLLOWER
I20260925 10:28:49.807803 28327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:49.808050 28327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:28:49.808172 28327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:28:49.808369 28327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:28:49.812393 28327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:49.812906 28327 leader_election.cc:302] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ae6424aad74bf9a68f489b6f26f399; no voters:
I20260925 10:28:49.813635 28327 leader_election.cc:288] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:28:49.813818 28331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:28:49.814971 28331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [term 3 LEADER]: Becoming Leader. State: Replica: 65ae6424aad74bf9a68f489b6f26f399, State: Running, Role: LEADER
I20260925 10:28:49.815562 28331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } }
I20260925 10:28:49.815928 28327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:28:49.818033 28333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ae6424aad74bf9a68f489b6f26f399. Latest consensus state: current_term: 3 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:49.818405 28333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:49.818171 28332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "65ae6424aad74bf9a68f489b6f26f399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ae6424aad74bf9a68f489b6f26f399" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 46803 } } }
I20260925 10:28:49.818581 28332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae6424aad74bf9a68f489b6f26f399 [sys.catalog]: This master's current role is: LEADER
I20260925 10:28:49.830264 28338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:28:49.840556 28338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:28:49.842247 28338 catalog_manager.cc:1259] Loaded cluster ID: 15b0e781d01c41eaaaad02b3a66a07cc
I20260925 10:28:49.842446 28338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:28:49.844970 28338 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7630a879ad google::LogMessage::Fail() at ??:0
@ 0x7f7630a8a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7630a874de google::LogMessage::Flush() at ??:0
@ 0x7f7630a88a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f76464ba512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f76464b7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f764653f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f764653f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260925 10:28:49.904604 28351 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f76336d3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76336d691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f76336d6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f76336b1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f763698b6db start_thread at ??:0
@ 0x7f762f92161f clone at ??:0
W20260925 10:28:50.172439 28329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:50.172755 28329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:50.172938 28329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:50.180460 28329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:50.180579 28329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:50.180796 28329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:28:50.190898 28329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:36175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=41941
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:50.192267 28329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:50.193729 28329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:50.207475 28371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:50.207574 28368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:50.207609 28369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:50.208204 28329 server_base.cc:1034] running on GCE node
I20260925 10:28:50.208828 28329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:50.209911 28329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:50.211148 28329 hybrid_clock.cc:648] HybridClock initialized: now 1790332130211085 us; error 57 us; skew 500 ppm
I20260925 10:28:50.211498 28329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:50 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:50.214871 28329 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:28:50.216334 28329 webserver.cc:460] Webserver started at http://127.12.248.65:41941/ using document root <none> and password file <none>
I20260925 10:28:50.217017 28329 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:50.217121 28329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:50.222604 28329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:28:50.225409 28378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:50.226874 28329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:28:50.227065 28329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "2f32a89e2294427da7ebfd14dc257ec4"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "a4780896b7d1dece83782a8d7bd58df1"
server_key_iv: "5520e3d71781e1c326d88dc60b9ae53c"
server_key_version: "encryptionkey@0"
I20260925 10:28:50.228098 28329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:50.247746 28329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:50.252669 28329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:50.252981 28329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:50.254150 28329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:50.256211 28385 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 10:28:50.262476 28329 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:28:50.262640 28329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 10:28:50.262804 28329 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:28:50.266069 28329 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:28:50.266187 28329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:50.266566 28385 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Bootstrap starting.
I20260925 10:28:50.295634 28385 log.cc:824] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:28:50.305234 28329 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36175
I20260925 10:28:50.305256 28493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36175 every 8 connection(s)
I20260925 10:28:50.306985 28329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:28:50.311650 28385 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:28:50.312148 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.310051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:56661)
0925 10:28:50.310887 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.311182 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":381,"threads_started":1}
I20260925 10:28:50.312233 28385 tablet_bootstrap.cc:492] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Bootstrap complete.
W20260925 10:28:50.312858 28494 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:46803 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:46803: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
I20260925 10:28:50.313054 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28329
I20260925 10:28:50.313735 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.313378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:43087)
0925 10:28:50.313554 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.313628 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:28:50.313930 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:41443
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=37761
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:50.314030 28385 ts_tablet_manager.cc:1397] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Time spent bootstrapping tablet: real 0.048s user 0.030s sys 0.016s
I20260925 10:28:50.315308 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.314296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:60451)
0925 10:28:50.314967 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.315127 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":322,"threads_started":1}
W20260925 10:28:50.315567 28494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:28:50.324038 28385 raft_consensus.cc:348] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:50.324906 28385 raft_consensus.cc:729] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f32a89e2294427da7ebfd14dc257ec4, State: Initialized, Role: FOLLOWER
I20260925 10:28:50.325629 28385 consensus_queue.cc:260] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:50.327313 28385 ts_tablet_manager.cc:1428] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 10:28:50.327487 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.327018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:41929)
0925 10:28:50.327214 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.327323 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:28:50.328498 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.328159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:44295)
0925 10:28:50.328313 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.328389 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 10:28:50.656783 28498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:50.657143 28498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:50.657325 28498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:50.665478 28498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:50.665628 28498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:50.665828 28498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:28:50.676297 28498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:41443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=37761
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:50.677721 28498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:50.679050 28498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:50.692677 28505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:50.692691 28507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:50.693074 28504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:50.693558 28498 server_base.cc:1034] running on GCE node
I20260925 10:28:50.694152 28498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:50.695068 28498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:50.696271 28498 hybrid_clock.cc:648] HybridClock initialized: now 1790332130696227 us; error 33 us; skew 500 ppm
I20260925 10:28:50.696636 28498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:50 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:50.699859 28498 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:28:50.701355 28498 webserver.cc:460] Webserver started at http://127.12.248.66:37761/ using document root <none> and password file <none>
I20260925 10:28:50.702105 28498 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:50.702255 28498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:50.707615 28498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:28:50.710439 28514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:50.711840 28498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:50.712018 28498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "f74c7bf75095482a86d7ed6db4f3bd01"
format_stamp: "Formatted at 2026-09-25 10:28:44 on dist-test-slave-f9rl"
server_key: "7b4ad05bbb32e23f0a676bec5f72a4c7"
server_key_iv: "401389b645ab97ee23d2cd6fe8156934"
server_key_version: "encryptionkey@0"
I20260925 10:28:50.713007 28498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:50.734090 28498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:50.738096 28498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:50.738379 28498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:50.739414 28498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:50.740810 28498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:50.740937 28498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:50.741065 28498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:50.741150 28498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:50.774618 28498 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:41443
I20260925 10:28:50.774722 28627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:41443 every 8 connection(s)
I20260925 10:28:50.776125 28498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:28:50.780831 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.779065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:53959)
0925 10:28:50.779775 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.780010 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260925 10:28:50.780973 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28498
W20260925 10:28:50.781567 28628 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:46803 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:46803: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
I20260925 10:28:50.781678 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:38501
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:46803
--webserver_port=40017
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:28:50.782614 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.782070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:47679)
0925 10:28:50.782218 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.782296 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:28:50.783595 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:50.783194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:57979)
0925 10:28:50.783343 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:50.783487 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 10:28:50.783808 28628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:28:51.117292 28631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:28:51.117687 28631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:28:51.117870 28631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:28:51.125218 28631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:28:51.125377 28631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:28:51.125588 28631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:28:51.136258 28631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:38501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=40017
--tserver_master_addrs=127.12.248.126:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:28:51.137668 28631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:28:51.139067 28631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:28:51.152557 28636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:51.152624 28637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:28:51.152513 28639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:28:51.153267 28631 server_base.cc:1034] running on GCE node
I20260925 10:28:51.153934 28631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:28:51.154850 28631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:28:51.156055 28631 hybrid_clock.cc:648] HybridClock initialized: now 1790332131155992 us; error 54 us; skew 500 ppm
I20260925 10:28:51.156402 28631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:28:51 dist-test-slave-f9rl krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332131, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:28:51.159521 28631 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:28:51.160939 28631 webserver.cc:460] Webserver started at http://127.12.248.67:40017/ using document root <none> and password file <none>
I20260925 10:28:51.161773 28631 fs_manager.cc:359] Metadata directory not provided
I20260925 10:28:51.161935 28631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:28:51.167326 28631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:28:51.170115 28646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:51.171485 28631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:28:51.171663 28631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0134e7e11e1c40c79165d6b72883f70b"
format_stamp: "Formatted at 2026-09-25 10:28:45 on dist-test-slave-f9rl"
server_key: "74f0c84d16459b252dfa85a56dab12fd"
server_key_iv: "cf98e177628605eee10e1892ddd56cab"
server_key_version: "encryptionkey@0"
I20260925 10:28:51.172653 28631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:28:51.191895 28631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:28:51.195861 28631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:28:51.196146 28631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:28:51.197140 28631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:28:51.198508 28631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:28:51.198638 28631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:51.198777 28631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:28:51.198868 28631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:28:51.233247 28631 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:38501
I20260925 10:28:51.233304 28759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:38501 every 8 connection(s)
I20260925 10:28:51.234822 28631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:28:51.239733 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:51.237801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:40767)
0925 10:28:51.238569 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:51.238784 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
W20260925 10:28:51.240439 28760 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:46803 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:46803: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
I20260925 10:28:51.241271 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:51.240915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:48807)
0925 10:28:51.241069 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:51.241142 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 10:28:51.242221 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:51.241900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:58815)
0925 10:28:51.242032 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:51.242111 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:28:51.242503 28760 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:28:51.244916 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28631
I20260925 10:28:51.330353 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:51.329602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:57579)
0925 10:28:51.330051 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:51.330144 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":210,"threads_started":1}
I20260925 10:28:51.785142 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:51.784342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:60807)
0925 10:28:51.784799 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:51.784909 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260925 10:28:52.019956 28766 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:52.020237 28766 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:52.021867 28766 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
Sep 25 10:28:52 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
Sep 25 10:28:52 dist-test-slave-f9rl krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:28:52.061430 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.022614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:46673 (local address 127.12.248.66:41443)
0925 10:28:52.035291 (+ 12677us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:52.035313 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:52.035717 (+ 404us) server_negotiation.cc:407] Connection header received
0925 10:28:52.038326 (+ 2609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:52.038335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:52.038677 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:52.039039 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:52.042206 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.044336 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.047463 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.048609 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.048863 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:52.054470 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:52.054478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:52.054491 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:52.054530 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:52.057706 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:52.058481 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:52.058487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:52.058493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:52.058644 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:52.059107 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:52.059112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:52.059117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:52.059652 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:52.059841 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:52.060639 (+ 798us) server_negotiation.cc:299] Negotiation successful
0925 10:28:52.061019 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12415,"thread_start_us":12172,"threads_started":1}
I20260925 10:28:52.061506 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.022148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41443 (local address 127.12.248.65:46673)
0925 10:28:52.035470 (+ 13322us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:52.035539 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:52.036261 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:52.039434 (+ 3173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:52.039447 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:52.040782 (+ 1335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:52.041921 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:52.041954 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.044577 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.044581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:52.047286 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:52.047302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.048847 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.048856 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:52.049252 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:52.050302 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:52.050334 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:52.054183 (+ 3849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:52.057855 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:52.057859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:52.057861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:52.058300 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:52.058790 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:52.058794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:52.058796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:52.058980 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:52.059796 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:52.059804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:52.060101 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 10:28:52.060370 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 10:28:52.060959 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13139,"mutex_wait_us":448,"spinlock_wait_cycles":9984,"thread_start_us":13165,"threads_started":1}
I20260925 10:28:52.061589 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.022338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38501 (local address 127.12.248.65:52751)
0925 10:28:52.035641 (+ 13303us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:52.035666 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:28:52.036237 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:28:52.039651 (+ 3414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:28:52.039657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:28:52.040988 (+ 1331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:28:52.041921 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:52.041954 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.044573 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.044579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:52.047285 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:28:52.047302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.048864 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.048868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:28:52.049248 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:52.050302 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:28:52.050342 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:28:52.053876 (+ 3534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:28:52.057641 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:52.057652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:52.057672 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:52.058261 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:52.058748 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:28:52.058753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:28:52.058759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:28:52.058992 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:28:52.059819 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:28:52.059823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:28:52.060093 (+ 270us) client_negotiation.cc:769] Sending connection context
0925 10:28:52.060371 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 10:28:52.061113 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13160,"mutex_wait_us":730,"spinlock_wait_cycles":43392,"thread_start_us":13231,"threads_started":1}
I20260925 10:28:52.061477 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.022438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:52751 (local address 127.12.248.67:38501)
0925 10:28:52.035805 (+ 13367us) server_negotiation.cc:206] Beginning negotiation
0925 10:28:52.035822 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:28:52.035861 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:28:52.038205 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:28:52.038216 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:28:52.038659 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:28:52.039082 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:28:52.042206 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.044340 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.047463 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:28:52.048688 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:28:52.048863 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:28:52.054222 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:28:52.054232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:28:52.054246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:28:52.054291 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:28:52.057408 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:52.058419 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:52.058429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:52.058438 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:52.058627 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:28:52.059141 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:28:52.059145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:28:52.059147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:28:52.059650 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:28:52.059852 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:28:52.060617 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 10:28:52.061029 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13167,"thread_start_us":13305,"threads_started":1}
W20260925 10:28:52.065310 28381 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
W20260925 10:28:52.065663 28382 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:52.065840 28382 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4; no voters: 0134e7e11e1c40c79165d6b72883f70b, f74c7bf75095482a86d7ed6db4f3bd01
I20260925 10:28:52.066334 28766 raft_consensus.cc:2726] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:28:52.243990 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.243107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:41613)
0925 10:28:52.243593 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:52.243764 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":223,"threads_started":1}
I20260925 10:28:52.331954 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.331442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:52005)
0925 10:28:52.331682 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:52.331783 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 10:28:52.786746 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:52.786055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:59843)
0925 10:28:52.786504 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:52.786587 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260925 10:28:53.245747 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:53.244971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:60565)
0925 10:28:53.245408 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:53.245590 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260925 10:28:53.334105 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:53.332996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:52745)
0925 10:28:53.333713 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:53.333825 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":296,"threads_started":1}
I20260925 10:28:53.788537 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:53.787669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:49027)
0925 10:28:53.788206 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:53.788319 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":311,"threads_started":1}
I20260925 10:28:53.827907 28776 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:53.828135 28776 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:53.828874 28776 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
W20260925 10:28:53.829816 28381 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
W20260925 10:28:53.830021 28382 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:53.830149 28382 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4; no voters: 0134e7e11e1c40c79165d6b72883f70b, f74c7bf75095482a86d7ed6db4f3bd01
I20260925 10:28:53.830422 28776 raft_consensus.cc:2726] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:28:54.247370 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:54.246665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:42305)
0925 10:28:54.247113 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:54.247199 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260925 10:28:54.335779 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:54.335067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:40317)
0925 10:28:54.335504 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:54.335609 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260925 10:28:54.790223 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:54.789528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:60415)
0925 10:28:54.789940 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:54.790044 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
I20260925 10:28:55.249058 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:55.248332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:60495)
0925 10:28:55.248793 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:55.248873 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":246,"threads_started":1}
I20260925 10:28:55.337729 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:55.336901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:60741)
0925 10:28:55.337379 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:55.337536 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":252,"threads_started":1}
I20260925 10:28:55.791864 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:55.791169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:46925)
0925 10:28:55.791606 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:55.791693 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260925 10:28:56.250775 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:56.250038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:60451)
0925 10:28:56.250514 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:56.250609 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260925 10:28:56.339542 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:56.338680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:33809)
0925 10:28:56.339192 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:56.339284 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
I20260925 10:28:56.647683 28785 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:56.647948 28785 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:56.648804 28785 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
W20260925 10:28:56.649856 28381 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
W20260925 10:28:56.650066 28382 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:56.650168 28382 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4; no voters: 0134e7e11e1c40c79165d6b72883f70b, f74c7bf75095482a86d7ed6db4f3bd01
I20260925 10:28:56.650447 28785 raft_consensus.cc:2726] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:28:56.793706 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:56.792801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:34767)
0925 10:28:56.793364 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:56.793491 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":338,"threads_started":1}
I20260925 10:28:57.252518 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:57.251712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:51619)
0925 10:28:57.252262 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:57.252344 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":307,"threads_started":1}
I20260925 10:28:57.341598 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:57.340636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:55903)
0925 10:28:57.341241 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:57.341330 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":301,"threads_started":1}
I20260925 10:28:57.795461 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:57.794680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:41905)
0925 10:28:57.795189 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:57.795281 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":230,"threads_started":1}
I20260925 10:28:58.254418 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:58.253615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:34007)
0925 10:28:58.254134 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:58.254228 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":231,"threads_started":1}
I20260925 10:28:58.343308 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:58.342474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:42699)
0925 10:28:58.343029 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:58.343130 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":249,"threads_started":1}
I20260925 10:28:58.797139 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:58.796429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:39573)
0925 10:28:58.796862 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:58.796943 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260925 10:28:59.256189 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:59.255431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:46431)
0925 10:28:59.255916 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:59.256002 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
I20260925 10:28:59.345165 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:59.344402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:54875)
0925 10:28:59.344869 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:59.344957 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":232,"threads_started":1}
I20260925 10:28:59.727175 28795 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:28:59.727497 28795 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:28:59.728375 28795 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
W20260925 10:28:59.729367 28381 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
W20260925 10:28:59.729602 28382 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:28:59.729696 28382 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4; no voters: 0134e7e11e1c40c79165d6b72883f70b, f74c7bf75095482a86d7ed6db4f3bd01
I20260925 10:28:59.729913 28795 raft_consensus.cc:2726] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:28:59.798803 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:28:59.798032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:35617)
0925 10:28:59.798557 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 10:28:59.798646 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":238,"threads_started":1}
I20260925 10:29:00.257903 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:00.257148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:33557)
0925 10:29:00.257653 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:00.257748 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":221,"threads_started":1}
I20260925 10:29:00.347146 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:00.346414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:48835)
0925 10:29:00.346887 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:00.346985 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260925 10:29:00.800540 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:00.799743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:44105)
0925 10:29:00.800239 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:00.800326 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":277,"threads_started":1}
I20260925 10:29:01.259716 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:01.258915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:41333)
0925 10:29:01.259442 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:01.259530 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260925 10:29:01.349103 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:01.348282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:54777)
0925 10:29:01.348800 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:01.348896 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":284,"threads_started":1}
I20260925 10:29:01.802263 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:01.801496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:41607)
0925 10:29:01.801980 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:01.802077 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260925 10:29:02.261386 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:02.260685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:44247)
0925 10:29:02.261136 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:02.261221 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260925 10:29:02.350816 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:02.350039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:59367)
0925 10:29:02.350539 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:02.350629 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260925 10:29:02.803908 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:02.803166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:48297)
0925 10:29:02.803654 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:02.803738 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":242,"threads_started":1}
I20260925 10:29:03.084653 28807 raft_consensus.cc:482] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:29:03.084847 28807 raft_consensus.cc:504] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0134e7e11e1c40c79165d6b72883f70b" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38501 } } peers { permanent_uuid: "2f32a89e2294427da7ebfd14dc257ec4" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36175 } } peers { permanent_uuid: "f74c7bf75095482a86d7ed6db4f3bd01" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41443 } }
I20260925 10:29:03.085784 28807 leader_election.cc:288] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501), f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443)
W20260925 10:29:03.086859 28381 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0134e7e11e1c40c79165d6b72883f70b (127.12.248.67:38501): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
W20260925 10:29:03.087061 28382 leader_election.cc:341] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f74c7bf75095482a86d7ed6db4f3bd01 (127.12.248.66:41443): Not found: Tablet not found: 7d88628ac95d4f5e835ddd6afe1e1d62
I20260925 10:29:03.087160 28382 leader_election.cc:302] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f32a89e2294427da7ebfd14dc257ec4; no voters: 0134e7e11e1c40c79165d6b72883f70b, f74c7bf75095482a86d7ed6db4f3bd01
I20260925 10:29:03.087379 28807 raft_consensus.cc:2726] T 7d88628ac95d4f5e835ddd6afe1e1d62 P 2f32a89e2294427da7ebfd14dc257ec4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:29:03.263230 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:03.262435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:48683)
0925 10:29:03.262925 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:03.263043 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":258,"threads_started":1}
I20260925 10:29:03.352641 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:03.351821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:58501)
0925 10:29:03.352310 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:03.352421 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260925 10:29:03.806129 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:03.805027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:57579)
0925 10:29:03.805744 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:03.805876 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":414,"threads_started":1}
I20260925 10:29:04.265060 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:04.264293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:44833)
0925 10:29:04.264779 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:04.264886 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
I20260925 10:29:04.354473 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:04.353737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:46645)
0925 10:29:04.354204 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:04.354294 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260925 10:29:04.807889 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:04.807106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:55287)
0925 10:29:04.807605 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:04.807692 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":236,"threads_started":1}
I20260925 10:29:05.266793 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:05.266024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:53523)
0925 10:29:05.266505 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:05.266592 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260925 10:29:05.356480 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:05.355752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.65:51607)
0925 10:29:05.356209 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:05.356302 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":218,"threads_started":1}
I20260925 10:29:05.809926 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:05.809140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.66:36725)
0925 10:29:05.809664 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:05.809766 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":222,"threads_started":1}
I20260925 10:29:06.257350 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28329
I20260925 10:29:06.268512 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:06.267821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:46803 (local address 127.12.248.67:33749)
0925 10:29:06.268240 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:06.268332 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:46803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260925 10:29:06.279204 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:06.258782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.12.248.65:36175)
0925 10:29:06.259487 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:06.259498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:06.259530 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:29:06.259626 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:06.259638 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:06.259985 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:06.260182 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:06.262110 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.263138 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.264340 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.264922 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.265030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:06.273399 (+ 8369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:06.273415 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:06.273429 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:06.273529 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:06.276400 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.277130 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.277138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.277140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.277281 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.277763 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.277767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.277769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.278179 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:06.278423 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:06.278710 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:29:06.278915 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":328,"threads_started":1}
I20260925 10:29:06.280504 28489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:06.374748 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28329
I20260925 10:29:06.393826 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28498
I20260925 10:29:06.407524 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:06.394626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.12.248.66:41443)
0925 10:29:06.395128 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:06.395132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:06.395149 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:29:06.395214 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:06.395218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:06.395278 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:06.395376 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:06.396740 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.397662 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.399009 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.399433 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.399541 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:06.402231 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:06.402236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:06.402239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:06.402274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:06.404986 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.405711 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.405716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.405718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.405869 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.406230 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.406233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.406235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.406586 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:06.406800 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:06.407053 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:29:06.407217 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260925 10:29:06.408245 28623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:06.494053 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28498
I20260925 10:29:06.511559 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28631
I20260925 10:29:06.527089 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:06.512636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.12.248.67:38501)
0925 10:29:06.513291 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:06.513297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:06.513322 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:29:06.513387 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:06.513392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:06.513549 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:06.513649 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:06.515219 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.515999 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.517596 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:06.518110 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:06.518225 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:06.521088 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:06.521093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:06.521096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:06.521129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:06.524146 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.524919 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.524924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.524927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.525085 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:06.525429 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:06.525578 (+ 149us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:06.525586 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:06.525893 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:06.526140 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:06.526431 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:29:06.526621 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":369,"threads_started":1}
I20260925 10:29:06.527865 28755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:06.623776 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28631
2026-09-25T10:29:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24807 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:29:06 dist-test-slave-f9rl krb5kdc[28826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:29:06 dist-test-slave-f9rl krb5kdc[28826](info): set up 2 sockets
Sep 25 10:29:06 dist-test-slave-f9rl krb5kdc[28826](info): commencing operation
krb5kdc: starting...
W20260925 10:29:07.767601 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:29:07 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:29:08.016650 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:08.364770 28839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:08.365149 28839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:08.365319 28839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:08.373914 28839 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:29:08.374084 28839 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:29:08.374161 28839 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:29:08.374239 28839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:29:08.374349 28839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:08.374488 28839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:08.374583 28839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:29:08.374665 28839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:29:08.384665 28839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:08.386094 28839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:08.387493 28839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:08.400003 28844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:08.400048 28847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:08.400087 28845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:08.400972 28839 server_base.cc:1034] running on GCE node
I20260925 10:29:08.401942 28839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:08.403414 28839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:08.404664 28839 hybrid_clock.cc:648] HybridClock initialized: now 1790332148404590 us; error 50 us; skew 500 ppm
I20260925 10:29:08.405074 28839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:08 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332148, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:08.408704 28839 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:29:08.410359 28839 webserver.cc:460] Webserver started at http://127.12.248.126:35751/ using document root <none> and password file <none>
I20260925 10:29:08.411072 28839 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:08.411177 28839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:08.411471 28839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:29:08.414779 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "794dd97b6f5842669806dffedf61a5f8"
format_stamp: "Formatted at 2026-09-25 10:29:08 on dist-test-slave-f9rl"
server_key: "097d7ed0bca0129c05e298e3dd83c08d"
server_key_iv: "04c87bd4ed3e8e72503dc9b6f338ef3d"
server_key_version: "encryptionkey@0"
I20260925 10:29:08.415622 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "794dd97b6f5842669806dffedf61a5f8"
format_stamp: "Formatted at 2026-09-25 10:29:08 on dist-test-slave-f9rl"
server_key: "097d7ed0bca0129c05e298e3dd83c08d"
server_key_iv: "04c87bd4ed3e8e72503dc9b6f338ef3d"
server_key_version: "encryptionkey@0"
I20260925 10:29:08.421746 28839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 10:29:08.425598 28854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:08.427119 28839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:29:08.427335 28839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "794dd97b6f5842669806dffedf61a5f8"
format_stamp: "Formatted at 2026-09-25 10:29:08 on dist-test-slave-f9rl"
server_key: "097d7ed0bca0129c05e298e3dd83c08d"
server_key_iv: "04c87bd4ed3e8e72503dc9b6f338ef3d"
server_key_version: "encryptionkey@0"
I20260925 10:29:08.427549 28839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:08.479455 28839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:08.484098 28839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:08.484408 28839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:08.503976 28839 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:40303
I20260925 10:29:08.503995 28905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:40303 every 8 connection(s)
I20260925 10:29:08.505343 28839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:29:08.509109 28906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:08.511775 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28839
I20260925 10:29:08.511984 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:29:08.512840 13281 external_mini_cluster.cc:1428] Setting key 235754fa968a38b62fc8b2c9f7a9eaa7
I20260925 10:29:08.522109 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap starting.
I20260925 10:29:08.526093 28906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:08.527493 28906 log.cc:824] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:29:08 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:08.530642 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: No bootstrap required, opened a new log
I20260925 10:29:08.538311 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:08.515135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.12.248.126:40303)
0925 10:29:08.516011 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:08.516023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:08.516077 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:29:08.517984 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:08.518007 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:08.518443 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:08.518789 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:08.520604 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:08.522089 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:08.524022 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:08.524769 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:08.524935 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:08.530195 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:08.530203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:08.530218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:08.530254 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:08.533767 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:08.534692 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:08.534698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:08.534705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:08.534887 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:08.535349 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:08.535352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:08.535354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:08.535972 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:08.536222 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:08.536685 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 10:29:08.537102 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":426,"threads_started":1}
I20260925 10:29:08.540422 28906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:08.540814 28906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:08.540946 28906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Initialized, Role: FOLLOWER
I20260925 10:29:08.541630 28906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:08.541952 28906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:29:08.542119 28906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:29:08.542299 28906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:08.544775 28906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:08.545331 28906 leader_election.cc:302] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794dd97b6f5842669806dffedf61a5f8; no voters:
I20260925 10:29:08.546051 28906 leader_election.cc:288] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:29:08.546260 28911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:29:08.546978 28911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Running, Role: LEADER
I20260925 10:29:08.547556 28911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:08.548199 28906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:29:08.550997 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:08.550993 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794dd97b6f5842669806dffedf61a5f8. Latest consensus state: current_term: 1 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:08.551414 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:08.551455 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:08.555011 28919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:29:08.562080 28919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:29:08.570946 28919 catalog_manager.cc:1347] Generated new cluster ID: bb828e79c7cc46699c3417cc4a93ed7b
I20260925 10:29:08.571079 28919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:29:08.579371 28919 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:29:08.581270 28919 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:29:08.589685 28919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Generated new TSK 0
I20260925 10:29:08.590821 28919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:29:08.649289 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:08.991796 28932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:08.992095 28932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:08.992261 28932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:08.999943 28932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:09.000063 28932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:09.000290 28932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:29:09.010624 28932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:09.012053 28932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:09.013512 28932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:09.027410 28937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:09.027498 28938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:09.027450 28940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:09.028789 28932 server_base.cc:1034] running on GCE node
I20260925 10:29:09.029505 28932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:09.030540 28932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:09.031837 28932 hybrid_clock.cc:648] HybridClock initialized: now 1790332149031753 us; error 54 us; skew 500 ppm
I20260925 10:29:09.032269 28932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:09 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:09.035902 28932 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:29:09.037513 28932 webserver.cc:460] Webserver started at http://127.12.248.65:36853/ using document root <none> and password file <none>
I20260925 10:29:09.038234 28932 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:09.038334 28932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:09.038594 28932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:29:09.041477 28932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "8937859fbd70412bbd3f0fe98af3daec"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "59211fccb3015c6d063dd8fcd518af9a"
server_key_iv: "7bf48736cd7b39ab8bd63a6a92143bd2"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.042215 28932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "8937859fbd70412bbd3f0fe98af3daec"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "59211fccb3015c6d063dd8fcd518af9a"
server_key_iv: "7bf48736cd7b39ab8bd63a6a92143bd2"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.049057 28932 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260925 10:29:09.053977 28947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.055466 28932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:29:09.055657 28932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "8937859fbd70412bbd3f0fe98af3daec"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "59211fccb3015c6d063dd8fcd518af9a"
server_key_iv: "7bf48736cd7b39ab8bd63a6a92143bd2"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.055866 28932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:09.080217 28932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:09.084444 28932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:09.084757 28932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:09.085860 28932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:09.087270 28932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:09.087389 28932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.087549 28932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:09.087620 28932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.131271 28932 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:35361
I20260925 10:29:09.131359 29061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:35361 every 8 connection(s)
I20260925 10:29:09.132728 28932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:29:09.138829 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 28932
I20260925 10:29:09.139053 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:29:09.139719 13281 external_mini_cluster.cc:1428] Setting key 730b35e6992b76472c17f2d6ff3285b0
Sep 25 10:29:09 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:09.154111 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:09.135551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49433 (local address 127.12.248.126:40303)
0925 10:29:09.136018 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:09.136023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:09.137138 (+ 1115us) server_negotiation.cc:407] Connection header received
0925 10:29:09.139149 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:09.139155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:09.139249 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:09.139370 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:09.140951 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.141808 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.143372 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.143778 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.143917 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:09.147583 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:09.147588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:09.147591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:09.147623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:09.150718 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:09.151686 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:09.151690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:09.151693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:09.151813 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:09.152321 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:09.152324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:09.152325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:09.152600 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:09.152709 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:09.153679 (+ 970us) server_negotiation.cc:299] Negotiation successful
0925 10:29:09.153847 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260925 10:29:09.155699 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:09.136204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:49433)
0925 10:29:09.136969 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:09.137008 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:09.138837 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:09.139621 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:09.139629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:09.140210 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:09.140778 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:09.140795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.141976 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.141981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:09.143213 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:09.143225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.144012 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.144021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:09.144284 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:09.144994 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:09.145016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:09.147331 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:09.150866 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:09.150878 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:09.150899 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:09.151561 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:09.151930 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:09.151937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:09.151944 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:09.152209 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:09.152732 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:09.152742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:09.153229 (+ 487us) client_negotiation.cc:769] Sending connection context
0925 10:29:09.153600 (+ 371us) client_negotiation.cc:240] Negotiation successful
0925 10:29:09.154186 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":308,"threads_started":1}
I20260925 10:29:09.157363 29062 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:09.157732 29062 heartbeater.cc:464] Registering TS with master...
I20260925 10:29:09.158530 29062 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:09.160799 28871 ts_manager.cc:194] Registered new tserver with Master: 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361)
I20260925 10:29:09.162386 28871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:49433
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:29:09.197743 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:09.536973 29068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:09.537315 29068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:09.537555 29068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:09.544874 29068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:09.545034 29068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:09.545225 29068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:29:09.555572 29068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:09.556949 29068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:09.558673 29068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:09.572608 29073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:09.572564 29074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:09.572638 29076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:09.573247 29068 server_base.cc:1034] running on GCE node
I20260925 10:29:09.573972 29068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:09.574925 29068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:09.576155 29068 hybrid_clock.cc:648] HybridClock initialized: now 1790332149576108 us; error 41 us; skew 500 ppm
I20260925 10:29:09.576473 29068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:09 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:09.579759 29068 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:29:09.581256 29068 webserver.cc:460] Webserver started at http://127.12.248.66:36137/ using document root <none> and password file <none>
I20260925 10:29:09.582011 29068 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:09.582162 29068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:09.582475 29068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:29:09.585529 29068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "b612219a067c47f3aa592388f312fc5d"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "c44d0e76ee58e43c3fc3051955113f2a"
server_key_iv: "64e4e4c463831c90a51c6b7e751326b9"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.586340 29068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "b612219a067c47f3aa592388f312fc5d"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "c44d0e76ee58e43c3fc3051955113f2a"
server_key_iv: "64e4e4c463831c90a51c6b7e751326b9"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.591674 29068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:29:09.595222 29083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.596506 29068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:29:09.596692 29068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "b612219a067c47f3aa592388f312fc5d"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "c44d0e76ee58e43c3fc3051955113f2a"
server_key_iv: "64e4e4c463831c90a51c6b7e751326b9"
server_key_version: "encryptionkey@0"
I20260925 10:29:09.596890 29068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:09.617544 29068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:09.621883 29068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:09.622248 29068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:09.623317 29068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:09.624598 29068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:09.624672 29068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.624758 29068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:09.624801 29068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:09.658828 29068 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:37617
I20260925 10:29:09.658926 29196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:37617 every 8 connection(s)
I20260925 10:29:09.660272 29068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:29:09.661643 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29068
I20260925 10:29:09.661847 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:29:09.662416 13281 external_mini_cluster.cc:1428] Setting key ee67245cc472ce1615e92f337f3b1500
Sep 25 10:29:09 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332149, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:09.681196 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:09.663263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:52077 (local address 127.12.248.126:40303)
0925 10:29:09.665769 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:09.665775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:09.665796 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:29:09.667032 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:09.667038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:09.667140 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:09.667260 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:09.668777 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.669649 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.671202 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.671646 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.671779 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:09.675397 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:09.675403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:09.675406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:09.675444 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:09.678060 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:09.678796 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:09.678803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:09.678806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:09.678944 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:09.679373 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:09.679378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:09.679382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:09.679661 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:09.679838 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:09.680525 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 10:29:09.680767 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2377,"thread_start_us":2264,"threads_started":1}
I20260925 10:29:09.682247 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:09.663896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:52077)
0925 10:29:09.664799 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:09.664844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:09.666758 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:09.667514 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:09.667521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:09.668030 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:09.668600 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:09.668618 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.669815 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.669821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:09.671053 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:09.671066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:09.671891 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:09.671898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:09.672151 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:09.672835 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:09.672864 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:09.675151 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:09.678218 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:09.678226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:09.678244 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:09.678676 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:09.679058 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:09.679065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:09.679070 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:09.679259 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:09.679795 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:09.679803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:09.680158 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 10:29:09.680449 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:29:09.681032 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":408,"threads_started":1}
I20260925 10:29:09.683867 29197 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:09.684199 29197 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
I20260925 10:29:09.684966 29197 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:09.686506 28871 ts_manager.cc:194] Registered new tserver with Master: b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
I20260925 10:29:09.687536 28871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:52077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:29:09.716805 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:10.055994 29203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:10.056317 29203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:10.056504 29203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:10.065551 29203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:10.065702 29203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:10.065904 29203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:29:10.077159 29203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:10.078663 29203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:10.080102 29203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:10.094882 29208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:10.095089 29209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:10.095577 29211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:10.096172 29203 server_base.cc:1034] running on GCE node
I20260925 10:29:10.096912 29203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:10.098163 29203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:10.099442 29203 hybrid_clock.cc:648] HybridClock initialized: now 1790332150099396 us; error 52 us; skew 500 ppm
I20260925 10:29:10.099819 29203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:10 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332150, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:10.103502 29203 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:29:10.105091 29203 webserver.cc:460] Webserver started at http://127.12.248.67:35779/ using document root <none> and password file <none>
I20260925 10:29:10.105923 29203 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:10.106109 29203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:10.106455 29203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:29:10.109774 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "0e1997c44576483c89593c685c0b6d48"
format_stamp: "Formatted at 2026-09-25 10:29:10 on dist-test-slave-f9rl"
server_key: "f4bb3719f3537827a93c14851832115b"
server_key_iv: "5950af662a475b88e07d5e8a64115ff4"
server_key_version: "encryptionkey@0"
I20260925 10:29:10.110632 29203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "0e1997c44576483c89593c685c0b6d48"
format_stamp: "Formatted at 2026-09-25 10:29:10 on dist-test-slave-f9rl"
server_key: "f4bb3719f3537827a93c14851832115b"
server_key_iv: "5950af662a475b88e07d5e8a64115ff4"
server_key_version: "encryptionkey@0"
I20260925 10:29:10.116611 29203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 10:29:10.120661 29218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:10.122381 29203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:29:10.122689 29203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0e1997c44576483c89593c685c0b6d48"
format_stamp: "Formatted at 2026-09-25 10:29:10 on dist-test-slave-f9rl"
server_key: "f4bb3719f3537827a93c14851832115b"
server_key_iv: "5950af662a475b88e07d5e8a64115ff4"
server_key_version: "encryptionkey@0"
I20260925 10:29:10.122924 29203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:10.154923 29203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:10.159714 29203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:10.160135 29203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:10.161187 29203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:10.162755 29203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:10.162837 29203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:10.162936 29203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:10.162992 29203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:10.165735 29062 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:10.199426 29203 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:36031
I20260925 10:29:10.199474 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:36031 every 8 connection(s)
I20260925 10:29:10.200904 29203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:29:10.210984 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29203
I20260925 10:29:10.211184 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:29:10.211802 13281 external_mini_cluster.cc:1428] Setting key de911d33d979520d83163eaf32183b71
Sep 25 10:29:10 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332150, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:10.222127 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.203709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:46503 (local address 127.12.248.126:40303)
0925 10:29:10.204158 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.204163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.205217 (+ 1054us) server_negotiation.cc:407] Connection header received
0925 10:29:10.207099 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.207104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.207198 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.207317 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:10.208799 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.209688 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.211758 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.212300 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.212457 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.216448 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:10.216454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:10.216458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:10.216496 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:10.219341 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.220037 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.220041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.220044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.220161 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.220546 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.220550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.220552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.220821 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:10.220951 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.221639 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.221817 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260925 10:29:10.223212 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.204353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:46503)
0925 10:29:10.205060 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.205102 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.206831 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.207544 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.207555 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.208061 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:10.208626 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.208646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.209868 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.209872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.211581 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.211595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.212523 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.212528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.212857 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.213732 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:10.213764 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:10.216229 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:10.219473 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:10.219480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:10.219494 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:10.219916 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:10.220260 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:10.220263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:10.220266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:10.220444 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:10.220927 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:10.220933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:10.221265 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.221542 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.221938 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":279,"threads_started":1}
I20260925 10:29:10.224747 29332 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:10.225131 29332 heartbeater.cc:464] Registering TS with master...
I20260925 10:29:10.226186 29332 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:10.228133 28870 ts_manager.cc:194] Registered new tserver with Master: 0e1997c44576483c89593c685c0b6d48 (127.12.248.67:36031)
I20260925 10:29:10.229115 28870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:46503
I20260925 10:29:10.240881 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:29:10.260449 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.245734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.12.248.126:40303)
0925 10:29:10.245995 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.246002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.246259 (+ 257us) server_negotiation.cc:407] Connection header received
0925 10:29:10.246414 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.246418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.246513 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.246602 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:10.248105 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.249040 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.250450 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.250919 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.251028 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.253902 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:10.253907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:10.253910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:10.253949 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:10.256811 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.257852 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.257856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.257859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.258122 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.258686 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.258694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.258699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.259205 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:10.259469 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.259876 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.260073 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 10:29:10.265225 28870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:29:10.272580 28870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:29:10.311668 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.285620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.12.248.67:36031)
0925 10:29:10.303300 (+ 17680us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.303309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.303347 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:29:10.303450 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.303459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.303723 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.303951 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:10.304722 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306294 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.308972 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.310338 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.310509 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.310782 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.311078 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.311260 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17469,"thread_start_us":17289,"threads_started":1}
I20260925 10:29:10.311893 29345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.285315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:36031 (local address 127.0.0.1:53462)
0925 10:29:10.300545 (+ 15230us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.300568 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.300688 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.303956 (+ 3268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.303961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.303987 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:10.304518 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.304530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306450 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.306456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.308761 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.308779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.310574 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.310581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.310812 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.310874 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.311030 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.311253 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15091,"mutex_wait_us":171,"thread_start_us":15448,"threads_started":1}
I20260925 10:29:10.312450 29346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.285272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:56618)
0925 10:29:10.300047 (+ 14775us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.300143 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.300438 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.304296 (+ 3858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.304303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.304343 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:10.304819 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.304833 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306912 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.306920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.309279 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.309293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.311440 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.311446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.311645 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.311781 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.311864 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.311992 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14326,"mutex_wait_us":82,"thread_start_us":15418,"threads_started":1}
I20260925 10:29:10.312935 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.287947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37617 (local address 127.0.0.1:38446)
0925 10:29:10.302122 (+ 14175us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.302146 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.302269 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.303484 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.303491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.303523 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:10.303970 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.303999 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306256 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.306261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.309185 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.309202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.311416 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.311426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.311644 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.311777 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.312110 (+ 333us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.312252 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14028,"mutex_wait_us":233,"thread_start_us":13854,"threads_started":1}
I20260925 10:29:10.313410 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.285581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.12.248.65:35361)
0925 10:29:10.303690 (+ 18109us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.303696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.303720 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:29:10.303803 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.303807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.304026 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.304266 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:10.305020 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306742 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.309481 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.310895 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.311036 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.311256 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.312276 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.312471 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17954,"thread_start_us":17801,"threads_started":1}
I20260925 10:29:10.315944 29266 tablet_service.cc:1467] Processing CreateTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e (DEFAULT_TABLE table=test-table [id=a48543389c85484599347f476a0d3ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:10.313311 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.288457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.12.248.66:37617)
0925 10:29:10.301630 (+ 13173us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.301640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.302408 (+ 768us) server_negotiation.cc:407] Connection header received
0925 10:29:10.302538 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.302547 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.302896 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.303184 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:10.304220 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.306101 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.309386 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.310895 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.311041 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.311221 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.312679 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.312857 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12986,"thread_start_us":12841,"threads_started":1}
I20260925 10:29:10.316459 28996 tablet_service.cc:1467] Processing CreateTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e (DEFAULT_TABLE table=test-table [id=a48543389c85484599347f476a0d3ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:10.317509 29266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b5f19707c479c90188d4cc1fa9a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:10.317955 28996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b5f19707c479c90188d4cc1fa9a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:10.320444 29131 tablet_service.cc:1467] Processing CreateTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e (DEFAULT_TABLE table=test-table [id=a48543389c85484599347f476a0d3ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:10.321880 29131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99b5f19707c479c90188d4cc1fa9a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:10.329066 29351 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Bootstrap starting.
I20260925 10:29:10.332769 29351 tablet_bootstrap.cc:654] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:10.333896 29351 log.cc:824] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:10.336886 29351 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: No bootstrap required, opened a new log
I20260925 10:29:10.337186 29351 ts_tablet_manager.cc:1397] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:29:10.337466 29354 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Bootstrap starting.
I20260925 10:29:10.337807 29352 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Bootstrap starting.
I20260925 10:29:10.341552 29354 tablet_bootstrap.cc:654] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:10.341552 29352 tablet_bootstrap.cc:654] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:10.343029 29354 log.cc:824] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:10.343012 29352 log.cc:824] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:10.346112 29351 raft_consensus.cc:348] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.346503 29351 raft_consensus.cc:374] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:10.346601 29352 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: No bootstrap required, opened a new log
I20260925 10:29:10.346604 29354 tablet_bootstrap.cc:492] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: No bootstrap required, opened a new log
I20260925 10:29:10.346603 29351 raft_consensus.cc:729] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1997c44576483c89593c685c0b6d48, State: Initialized, Role: FOLLOWER
I20260925 10:29:10.346920 29352 ts_tablet_manager.cc:1397] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 10:29:10.346920 29354 ts_tablet_manager.cc:1397] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 10:29:10.347254 29351 consensus_queue.cc:260] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.348380 29332 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:10.349216 29351 ts_tablet_manager.cc:1428] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Time spent starting tablet: real 0.012s user 0.005s sys 0.009s
I20260925 10:29:10.359323 29354 raft_consensus.cc:348] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.359340 29352 raft_consensus.cc:348] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.359892 29354 raft_consensus.cc:374] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:10.359895 29352 raft_consensus.cc:374] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:10.360018 29354 raft_consensus.cc:729] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b612219a067c47f3aa592388f312fc5d, State: Initialized, Role: FOLLOWER
I20260925 10:29:10.360024 29352 raft_consensus.cc:729] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8937859fbd70412bbd3f0fe98af3daec, State: Initialized, Role: FOLLOWER
I20260925 10:29:10.360750 29354 consensus_queue.cc:260] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.360750 29352 consensus_queue.cc:260] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.362957 29352 ts_tablet_manager.cc:1428] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260925 10:29:10.363171 29354 ts_tablet_manager.cc:1428] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260925 10:29:10.363261 29197 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:10.366057 29357 raft_consensus.cc:482] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:29:10.366370 29357 raft_consensus.cc:504] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.368363 29357 leader_election.cc:288] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361), b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
I20260925 10:29:10.376312 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.368652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:41931 (local address 127.12.248.65:35361)
0925 10:29:10.368864 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.368869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.369299 (+ 430us) server_negotiation.cc:407] Connection header received
0925 10:29:10.369479 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.369494 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.369577 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.369672 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:10.370206 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.371241 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.373858 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.375164 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.375338 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.375412 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.375912 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.376062 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:29:10.376308 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.369763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:59771 (local address 127.12.248.66:37617)
0925 10:29:10.369923 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.369927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.369942 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:29:10.369988 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.369991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.370041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.370109 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:10.370686 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.371584 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.374001 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.375258 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.375389 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.375469 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.375912 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.376051 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:29:10.376499 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.368567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.12.248.67:41931)
0925 10:29:10.369205 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.369221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.369323 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.369848 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.369853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.369888 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:10.370036 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.370044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.371486 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.371490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.373682 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.373703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.375362 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.375369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.375619 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.375737 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.375838 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.376046 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":94}
I20260925 10:29:10.376854 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.368895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37617 (local address 127.12.248.67:59771)
0925 10:29:10.369563 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:10.369581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:10.369702 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:10.370160 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:10.370167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:10.370196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:10.370535 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.370543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.371730 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.371735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.373840 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:10.373854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.375444 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.375449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:10.375698 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.375751 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:29:10.375838 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:29:10.376046 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":94,"thread_start_us":667,"threads_started":1}
I20260925 10:29:10.377553 29016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b5f19707c479c90188d4cc1fa9a8e" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8937859fbd70412bbd3f0fe98af3daec" is_pre_election: true
I20260925 10:29:10.377890 29151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b5f19707c479c90188d4cc1fa9a8e" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b612219a067c47f3aa592388f312fc5d" is_pre_election: true
I20260925 10:29:10.378172 29016 raft_consensus.cc:2454] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 0.
I20260925 10:29:10.378414 29151 raft_consensus.cc:2454] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 0.
I20260925 10:29:10.378937 29222 leader_election.cc:302] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1997c44576483c89593c685c0b6d48, 8937859fbd70412bbd3f0fe98af3daec; no voters:
I20260925 10:29:10.379295 29357 raft_consensus.cc:2781] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:29:10.379451 29357 raft_consensus.cc:482] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:29:10.379588 29357 raft_consensus.cc:3037] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:10.381870 29357 raft_consensus.cc:504] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.382543 29357 leader_election.cc:288] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 election: Requested vote from peers 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361), b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
I20260925 10:29:10.382944 29151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b5f19707c479c90188d4cc1fa9a8e" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b612219a067c47f3aa592388f312fc5d"
I20260925 10:29:10.382939 29016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99b5f19707c479c90188d4cc1fa9a8e" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8937859fbd70412bbd3f0fe98af3daec"
I20260925 10:29:10.383148 29151 raft_consensus.cc:3037] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:10.383148 29016 raft_consensus.cc:3037] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:29:10.385169 29063 tablet.cc:2369] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:29:10.386372 29016 raft_consensus.cc:2454] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 1.
I20260925 10:29:10.386372 29151 raft_consensus.cc:2454] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 1.
I20260925 10:29:10.386750 29222 leader_election.cc:302] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1997c44576483c89593c685c0b6d48, 8937859fbd70412bbd3f0fe98af3daec; no voters:
I20260925 10:29:10.387054 29357 raft_consensus.cc:2781] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:29:10.387513 29357 raft_consensus.cc:686] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1997c44576483c89593c685c0b6d48, State: Running, Role: LEADER
I20260925 10:29:10.387967 29357 consensus_queue.cc:237] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } }
I20260925 10:29:10.392793 28871 catalog_manager.cc:5495] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e1997c44576483c89593c685c0b6d48 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "0e1997c44576483c89593c685c0b6d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } health_report { overall_health: UNKNOWN } } }
W20260925 10:29:10.412492 29198 tablet.cc:2369] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:29:10.413353 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.407496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.12.248.67:36031)
0925 10:29:10.407713 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.407719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.407739 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:29:10.407970 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.407974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.408075 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.408232 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:29:10.408636 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.409346 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.410835 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.411254 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.411393 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.411945 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:29:10.412638 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:29:10.412802 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.412954 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.413059 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:29:10.422214 29151 raft_consensus.cc:1261] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Refusing update from remote peer 0e1997c44576483c89593c685c0b6d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:29:10.422214 29016 raft_consensus.cc:1261] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Refusing update from remote peer 0e1997c44576483c89593c685c0b6d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:29:10.423206 29361 consensus_queue.cc:1035] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:29:10.423581 29357 consensus_queue.cc:1035] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:29:10.450596 28870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:29:10.450860 28870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:29:10.454444 28870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Sending DeleteTablet for 3 replicas of tablet c99b5f19707c479c90188d4cc1fa9a8e
I20260925 10:29:10.455518 29266 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:10 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:53462
I20260925 10:29:10.455556 28996 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:10 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56618
I20260925 10:29:10.455765 29131 tablet_service.cc:1514] Processing DeleteTablet for tablet c99b5f19707c479c90188d4cc1fa9a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:10 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:38446
I20260925 10:29:10.457046 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28932
I20260925 10:29:10.466833 29375 tablet_replica.cc:330] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: stopping tablet replica
I20260925 10:29:10.467485 29375 raft_consensus.cc:2229] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:29:10.468158 29375 raft_consensus.cc:2258] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:10.470399 29375 ts_tablet_manager.cc:1905] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:10.470854 29376 tablet_replica.cc:330] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: stopping tablet replica
I20260925 10:29:10.471338 29376 raft_consensus.cc:2229] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:29:10.471865 29376 raft_consensus.cc:2258] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:10.473747 29374 tablet_replica.cc:330] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: stopping tablet replica
I20260925 10:29:10.474102 29376 ts_tablet_manager.cc:1905] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:10.479811 29375 ts_tablet_manager.cc:1918] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:10.480100 29374 raft_consensus.cc:2229] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:29:10.481024 29374 raft_consensus.cc:2258] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:10.481627 29376 ts_tablet_manager.cc:1918] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:10.481798 29376 log.cc:1196] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/c99b5f19707c479c90188d4cc1fa9a8e
Sep 25 10:29:10 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:29:10.482553 29376 ts_tablet_manager.cc:1939] T c99b5f19707c479c90188d4cc1fa9a8e P b612219a067c47f3aa592388f312fc5d: Deleting consensus metadata
I20260925 10:29:10.484051 29374 ts_tablet_manager.cc:1905] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:10.484802 28856 catalog_manager.cc:4841] TS b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617): tablet c99b5f19707c479c90188d4cc1fa9a8e (table test-table [id=a48543389c85484599347f476a0d3ad5]) successfully deleted
I20260925 10:29:10.487341 29375 log.cc:1196] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/c99b5f19707c479c90188d4cc1fa9a8e
I20260925 10:29:10.487922 29375 ts_tablet_manager.cc:1939] T c99b5f19707c479c90188d4cc1fa9a8e P 8937859fbd70412bbd3f0fe98af3daec: Deleting consensus metadata
I20260925 10:29:10.490145 28858 catalog_manager.cc:4841] TS 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361): tablet c99b5f19707c479c90188d4cc1fa9a8e (table test-table [id=a48543389c85484599347f476a0d3ad5]) successfully deleted
I20260925 10:29:10.491289 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.458643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.12.248.65:35361)
0925 10:29:10.458884 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.458892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.470151 (+ 11259us) server_negotiation.cc:407] Connection header received
0925 10:29:10.470381 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.470393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.470484 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.470611 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:10.472940 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.474123 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.476419 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.477153 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.477262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.483565 (+ 6303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:10.483580 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:10.483598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:10.483643 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:10.487065 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.488100 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.488112 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.488119 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.488328 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.488860 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.488864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.488868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.489348 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:10.489674 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.490659 (+ 985us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.491023 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:29:10.492015 29057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:10.494097 29374 ts_tablet_manager.cc:1918] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:10.494361 29374 log.cc:1196] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/c99b5f19707c479c90188d4cc1fa9a8e
I20260925 10:29:10.495217 29374 ts_tablet_manager.cc:1939] T c99b5f19707c479c90188d4cc1fa9a8e P 0e1997c44576483c89593c685c0b6d48: Deleting consensus metadata
I20260925 10:29:10.497689 28855 catalog_manager.cc:4841] TS 0e1997c44576483c89593c685c0b6d48 (127.12.248.67:36031): tablet c99b5f19707c479c90188d4cc1fa9a8e (table test-table [id=a48543389c85484599347f476a0d3ad5]) successfully deleted
I20260925 10:29:10.596854 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28932
I20260925 10:29:10.615737 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29068
Sep 25 10:29:10 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:29:10.634814 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.616704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.12.248.66:37617)
0925 10:29:10.616940 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.616947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.616971 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:29:10.617047 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.617054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.617148 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.617294 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:10.618790 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.620134 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.621791 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.622573 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.622719 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.627942 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:10.627960 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:10.627976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:10.628028 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:10.631112 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.632240 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.632251 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.632256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.632427 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.632894 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.632900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.632904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.633516 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:10.633855 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.634154 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.634401 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:29:10.635648 29192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:10.725265 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29068
I20260925 10:29:10.751164 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29203
Sep 25 10:29:10 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:29:10.770212 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:10.752032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.12.248.67:36031)
0925 10:29:10.752244 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:10.752251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:10.752276 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:29:10.752351 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:10.752356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:10.752445 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:10.752579 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:10.754052 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.755428 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.756879 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:10.757714 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:10.757884 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:10.762475 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:10.762490 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:10.762510 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:10.762568 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:10.766921 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.767647 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.767657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.767661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.767874 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:10.768301 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:10.768307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:10.768311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:10.768805 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:10.769131 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:10.769428 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:29:10.769709 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:29:10.770990 29326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:10.894567 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29203
I20260925 10:29:10.914167 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 28839
I20260925 10:29:10.915308 28901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:11.028849 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 28839
I20260925 10:29:31.051009 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--webserver_interface=127.12.248.126
--webserver_port=35751
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:31.397635 29382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:31.398003 29382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:31.398190 29382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:31.405530 29382 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:29:31.405689 29382 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:29:31.405788 29382 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:29:31.405885 29382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:29:31.405967 29382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:31.406090 29382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:31.406199 29382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:29:31.406283 29382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:29:31.416313 29382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:31.417750 29382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:31.419157 29382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:31.443770 29391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:31.443783 29388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:31.443796 29389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:31.444582 29382 server_base.cc:1034] running on GCE node
I20260925 10:29:31.445557 29382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:31.447271 29382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:31.448531 29382 hybrid_clock.cc:648] HybridClock initialized: now 1790332171448455 us; error 50 us; skew 500 ppm
I20260925 10:29:31.448956 29382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:31 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332171, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:31.452764 29382 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:29:31.454547 29382 webserver.cc:460] Webserver started at http://127.12.248.126:35751/ using document root <none> and password file <none>
I20260925 10:29:31.455359 29382 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:31.455469 29382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:31.462037 29382 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:29:31.465205 29398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:31.466729 29382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:29:31.466876 29382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "794dd97b6f5842669806dffedf61a5f8"
format_stamp: "Formatted at 2026-09-25 10:29:08 on dist-test-slave-f9rl"
server_key: "097d7ed0bca0129c05e298e3dd83c08d"
server_key_iv: "04c87bd4ed3e8e72503dc9b6f338ef3d"
server_key_version: "encryptionkey@0"
I20260925 10:29:31.467913 29382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:31.483402 29382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:31.487730 29382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:31.487999 29382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:31.508325 29382 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:40303
I20260925 10:29:31.508526 29449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:40303 every 8 connection(s)
I20260925 10:29:31.510020 29382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:29:31.516451 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29382
I20260925 10:29:31.517558 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:35361
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=36853
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:29:31.519733 29450 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:29:31.525484 29450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap starting.
I20260925 10:29:31.548816 29450 log.cc:824] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:31.561626 29450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:29:31.562198 29450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap complete.
I20260925 10:29:31.571916 29450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:31.572940 29450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Initialized, Role: FOLLOWER
I20260925 10:29:31.573632 29450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:31.573887 29450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:29:31.574028 29450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:29:31.574231 29450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:29:31.578774 29450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:31.579322 29450 leader_election.cc:302] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794dd97b6f5842669806dffedf61a5f8; no voters:
I20260925 10:29:31.580054 29450 leader_election.cc:288] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:29:31.580188 29454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:29:31.581416 29454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 LEADER]: Becoming Leader. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Running, Role: LEADER
I20260925 10:29:31.582008 29454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:31.582559 29450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:29:31.584530 29456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794dd97b6f5842669806dffedf61a5f8. Latest consensus state: current_term: 2 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:31.584872 29456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:31.585291 29455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:31.585690 29455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:31.586024 29460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:29:31.605906 29460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:29:31.607985 29460 catalog_manager.cc:1259] Loaded cluster ID: bb828e79c7cc46699c3417cc4a93ed7b
I20260925 10:29:31.608331 29460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:29:31.613762 29460 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:29:31.615939 29460 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Loaded TSK: 0
I20260925 10:29:31.659587 29460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Generated new TSK 1
I20260925 10:29:31.660032 29460 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:29:31.918745 29452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:31.919087 29452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:31.919255 29452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:31.927070 29452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:31.927238 29452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:31.927413 29452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:29:31.937707 29452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=36853
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:31.939122 29452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:31.940536 29452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:31.956046 29477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:31.956279 29478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:31.956727 29480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:31.958003 29452 server_base.cc:1034] running on GCE node
I20260925 10:29:31.958770 29452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:31.960039 29452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:31.961308 29452 hybrid_clock.cc:648] HybridClock initialized: now 1790332171961234 us; error 57 us; skew 500 ppm
I20260925 10:29:31.961742 29452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:31 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332171, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:31.966003 29452 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:29:31.967888 29452 webserver.cc:460] Webserver started at http://127.12.248.65:36853/ using document root <none> and password file <none>
I20260925 10:29:31.968797 29452 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:31.968945 29452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:31.976112 29452 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:29:31.979666 29487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:31.981473 29452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:29:31.981649 29452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "8937859fbd70412bbd3f0fe98af3daec"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "59211fccb3015c6d063dd8fcd518af9a"
server_key_iv: "7bf48736cd7b39ab8bd63a6a92143bd2"
server_key_version: "encryptionkey@0"
I20260925 10:29:31.983037 29452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:32.017875 29452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:32.023135 29452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:32.023443 29452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:32.024670 29452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:32.026348 29452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:32.026470 29452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.026582 29452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:32.026652 29452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.061327 29452 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:35361
I20260925 10:29:32.061493 29599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:35361 every 8 connection(s)
I20260925 10:29:32.062865 29452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:29:32.067512 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29452
I20260925 10:29:32.068166 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:37617
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=36137
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:29:32 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332171, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:32.088397 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.066135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:58147)
0925 10:29:32.066979 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:32.067025 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:32.068965 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:32.070616 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:32.070623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:32.071199 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:32.071860 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:32.071880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.074510 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.074514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:32.075790 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:32.075803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.076870 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.076879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:32.077175 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:32.077893 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:32.077917 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:32.080103 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:32.083637 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:32.083646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:32.083662 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:32.084176 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:32.084632 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:32.084637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:32.084641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:32.084831 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:32.085657 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:32.085666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:32.086109 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 10:29:32.086430 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 10:29:32.087021 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":352,"threads_started":1}
I20260925 10:29:32.088389 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.065967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:58147 (local address 127.12.248.126:40303)
0925 10:29:32.066579 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:32.066592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:32.067191 (+ 599us) server_negotiation.cc:407] Connection header received
0925 10:29:32.069615 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:32.069635 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:32.070107 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:32.070502 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:32.072104 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.074324 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.075949 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.076610 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.076833 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:32.080399 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:32.080407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:32.080422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:32.080462 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:32.083476 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:32.084321 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:32.084328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:32.084335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:32.084496 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:32.084946 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:32.084952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:32.084955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:32.085504 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:32.085704 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:32.086675 (+ 971us) server_negotiation.cc:299] Negotiation successful
0925 10:29:32.087077 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260925 10:29:32.091740 29600 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:32.092068 29600 heartbeater.cc:464] Registering TS with master...
I20260925 10:29:32.092814 29600 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:32.095245 29415 ts_manager.cc:194] Registered new tserver with Master: 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361)
I20260925 10:29:32.097291 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:58147
W20260925 10:29:32.406498 29604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:32.406780 29604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:32.406874 29604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:32.414250 29604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:32.414362 29604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:32.414484 29604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:29:32.424067 29604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:37617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=36137
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:32.425474 29604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:32.426810 29604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:32.439859 29609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:32.439939 29610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:32.439965 29612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:32.440888 29604 server_base.cc:1034] running on GCE node
I20260925 10:29:32.441486 29604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:32.442454 29604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:32.443657 29604 hybrid_clock.cc:648] HybridClock initialized: now 1790332172443619 us; error 35 us; skew 500 ppm
I20260925 10:29:32.444065 29604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:32 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:32.447350 29604 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:29:32.448788 29604 webserver.cc:460] Webserver started at http://127.12.248.66:36137/ using document root <none> and password file <none>
I20260925 10:29:32.449539 29604 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:32.449695 29604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:32.455169 29604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 10:29:32.457906 29619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.459321 29604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:29:32.459518 29604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "b612219a067c47f3aa592388f312fc5d"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "c44d0e76ee58e43c3fc3051955113f2a"
server_key_iv: "64e4e4c463831c90a51c6b7e751326b9"
server_key_version: "encryptionkey@0"
I20260925 10:29:32.460544 29604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:32.486042 29604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:32.490854 29604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:32.491163 29604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:32.492254 29604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:32.493773 29604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:32.493877 29604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.494011 29604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:32.494119 29604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.527611 29604 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:37617
I20260925 10:29:32.527717 29733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:37617 every 8 connection(s)
I20260925 10:29:32.529547 29604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:29:32.534250 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29604
I20260925 10:29:32.534880 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:36031
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=35779
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:29:32 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:32.550990 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.532317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:46451 (local address 127.12.248.126:40303)
0925 10:29:32.532495 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:32.532500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:32.533781 (+ 1281us) server_negotiation.cc:407] Connection header received
0925 10:29:32.536329 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:32.536337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:32.536449 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:32.536571 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:32.538188 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.538906 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.540478 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.540919 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.541071 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:32.544989 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:32.544994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:32.544998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:32.545034 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:32.547596 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:32.548363 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:32.548367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:32.548370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:32.548497 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:32.548990 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:32.549000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:32.549002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:32.549314 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:32.549502 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:32.550390 (+ 888us) server_negotiation.cc:299] Negotiation successful
0925 10:29:32.550644 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:29:32.552094 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.532745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:46451)
0925 10:29:32.533598 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:32.533655 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:32.536029 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:32.536821 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:32.536830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:32.537346 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:32.538008 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:32.538026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.539069 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.539077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:32.540322 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:32.540334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:32.541158 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:32.541167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:32.541412 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:32.542413 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:32.542452 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:32.544734 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:32.547749 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:32.547758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:32.547777 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:32.548220 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:32.548651 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:32.548656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:32.548660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:32.548861 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:32.549474 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:32.549483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:32.549980 (+ 497us) client_negotiation.cc:769] Sending connection context
0925 10:29:32.550290 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 10:29:32.550869 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":349,"threads_started":1}
I20260925 10:29:32.553740 29734 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:32.554116 29734 heartbeater.cc:464] Registering TS with master...
I20260925 10:29:32.555099 29734 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:32.556874 29415 ts_manager.cc:194] Registered new tserver with Master: b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
I20260925 10:29:32.557744 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:46451
W20260925 10:29:32.869917 29737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:32.870216 29737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:32.870405 29737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:32.878198 29737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:32.878327 29737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:32.878520 29737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:29:32.888931 29737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:36031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=35779
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:32.890441 29737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:32.891793 29737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:32.905543 29742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:32.905668 29743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:32.905826 29745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:32.906410 29737 server_base.cc:1034] running on GCE node
I20260925 10:29:32.907052 29737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:32.908044 29737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:32.909263 29737 hybrid_clock.cc:648] HybridClock initialized: now 1790332172909212 us; error 41 us; skew 500 ppm
I20260925 10:29:32.909629 29737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:32 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:32.919986 29737 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:29:32.921772 29737 webserver.cc:460] Webserver started at http://127.12.248.67:35779/ using document root <none> and password file <none>
I20260925 10:29:32.922472 29737 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:32.922626 29737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:32.928177 29737 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:29:32.931051 29752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.932454 29737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:29:32.932637 29737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0e1997c44576483c89593c685c0b6d48"
format_stamp: "Formatted at 2026-09-25 10:29:10 on dist-test-slave-f9rl"
server_key: "f4bb3719f3537827a93c14851832115b"
server_key_iv: "5950af662a475b88e07d5e8a64115ff4"
server_key_version: "encryptionkey@0"
I20260925 10:29:32.933670 29737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:32.950469 29737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:32.954864 29737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:32.955163 29737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:32.956133 29737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:32.958046 29737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:32.958173 29737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.958283 29737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:32.958364 29737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:32.991685 29737 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:36031
I20260925 10:29:32.991739 29864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:36031 every 8 connection(s)
I20260925 10:29:32.993297 29737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:29:32.999434 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29737
Sep 25 10:29:33 dist-test-slave-f9rl krb5kdc[28826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:29:33.014035 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.995972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:51413 (local address 127.12.248.126:40303)
0925 10:29:32.996166 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:32.996170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:32.997316 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 10:29:32.999256 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:32.999262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:32.999355 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:32.999466 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.001394 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.002221 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.003758 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.004251 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.004385 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.007894 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.007899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.007902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.007933 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.010962 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.011704 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.011711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.011714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.011849 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.012285 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.012291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.012294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.012555 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.012710 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.013397 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.013650 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:29:33.014768 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.000420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.12.248.126:40303)
0925 10:29:33.000914 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.000919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.001246 (+ 327us) server_negotiation.cc:407] Connection header received
0925 10:29:33.001472 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.001479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.001548 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.001664 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.003207 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.004021 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.005639 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.006069 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.006223 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.008983 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.008987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.008990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.009028 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.011953 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.012689 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.012697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.012700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.012821 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.013206 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.013211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.013214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.013528 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.013830 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.014120 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.014424 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260925 10:29:33.015594 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:32.996452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:51413)
0925 10:29:32.997157 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:32.997194 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:32.998978 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:32.999723 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:32.999732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.000415 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:29:33.001218 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.001237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.002388 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.002393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.003613 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.003626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.004481 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.004488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.004729 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.005424 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:29:33.005502 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:29:33.007679 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:29:33.011126 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:33.011135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:33.011154 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:33.011582 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:33.011962 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:29:33.011968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:29:33.011971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:29:33.012168 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:29:33.012685 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:29:33.012693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:29:33.013038 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.013318 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.013893 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":278,"threads_started":1}
I20260925 10:29:33.016991 29865 heartbeater.cc:347] Connected to a master server at 127.12.248.126:40303
I20260925 10:29:33.017342 29865 heartbeater.cc:464] Registering TS with master...
I20260925 10:29:33.018100 29865 heartbeater.cc:511] Master 127.12.248.126:40303 requested a full tablet report, sending...
I20260925 10:29:33.019418 29414 ts_manager.cc:194] Registered new tserver with Master: 0e1997c44576483c89593c685c0b6d48 (127.12.248.67:36031)
I20260925 10:29:33.020253 29414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:51413
I20260925 10:29:33.029994 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:29:33.036616 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.031002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.12.248.126:40303)
0925 10:29:33.031216 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.031225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.031742 (+ 517us) server_negotiation.cc:407] Connection header received
0925 10:29:33.031917 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.031920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.032070 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.032195 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:29:33.032585 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.033289 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.034895 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.035310 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.035400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.035878 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:29:33.036109 (+ 231us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:29:33.036375 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52614: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:29:33.036748 29869 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52614: token expired
W20260925 10:29:33.037369 29339 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.248.126:40303)
I20260925 10:29:33.050782 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.037938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.12.248.126:40303)
0925 10:29:33.038111 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.038118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.038156 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:29:33.038227 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.038230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.038284 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.038359 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.039694 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.040562 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.041889 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.042288 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.042385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.045219 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.045225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.045229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.045268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.048207 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.048881 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.048885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.048887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.049016 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.049319 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.049322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.049324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.049761 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.049974 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.050326 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.050494 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:29:33.054167 29414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:29:33.061805 29414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:29:33.100605 29600 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:33.104274 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.074940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:46516)
0925 10:29:33.076163 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.076203 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:33.076397 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:33.094704 (+ 18307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:33.094714 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.094751 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:33.096580 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.096595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.098459 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.098470 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.101089 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.101104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.103168 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.103175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.103367 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.103432 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.103554 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.103765 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":18,"thread_start_us":4958,"threads_started":1}
I20260925 10:29:33.105116 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.12.248.67:36031)
0925 10:29:33.091848 (+ 15365us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.091863 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.091910 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:29:33.092058 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.092072 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.092430 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.092690 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:33.096394 (+ 3704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.098160 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.102012 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.103440 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.103545 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.103776 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.104549 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.104741 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15113,"thread_start_us":16478,"threads_started":1}
I20260925 10:29:33.105266 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.075089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37617 (local address 127.0.0.1:39574)
0925 10:29:33.084822 (+ 9733us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.084849 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:33.085002 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:33.095058 (+ 10056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:33.095064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.095098 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:33.096161 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.096174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.100017 (+ 3843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.100023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.102504 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.102516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.104009 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.104017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.104234 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.104313 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.104448 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.104595 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9551,"mutex_wait_us":186,"thread_start_us":12672,"threads_started":1}
I20260925 10:29:33.105628 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.075152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.12.248.65:35361)
0925 10:29:33.094010 (+ 18858us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.094020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.094059 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:29:33.094154 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.094160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.094420 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.094635 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:33.096779 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.098316 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.101319 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.102916 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.104766 (+ 1850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.105012 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.105084 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.105275 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18641,"thread_start_us":18500,"threads_started":1}
I20260925 10:29:33.106149 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.076215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:36031 (local address 127.0.0.1:56498)
0925 10:29:33.087486 (+ 11271us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.087512 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:33.087658 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:33.095011 (+ 7353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:33.095016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.095051 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:33.096124 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.096141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.098363 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.098371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.101836 (+ 3465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.101850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.104030 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.104043 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.104316 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.104353 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.105585 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.105753 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11088,"mutex_wait_us":941,"thread_start_us":8212,"threads_started":1}
I20260925 10:29:33.106360 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.075704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.12.248.66:37617)
0925 10:29:33.093177 (+ 17473us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.093189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.093225 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:29:33.093338 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.093346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.093628 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.095191 (+ 1563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:33.096434 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.099816 (+ 3382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.102671 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.103697 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.103873 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.104123 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.105872 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.106073 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17254,"thread_start_us":17527,"threads_started":1}
I20260925 10:29:33.109915 29535 tablet_service.cc:1467] Processing CreateTablet for tablet 298c98786a05403895ce392dbff6212c (DEFAULT_TABLE table=test-table [id=bd3c666a1f1c49c9917430b883568840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:33.109915 29800 tablet_service.cc:1467] Processing CreateTablet for tablet 298c98786a05403895ce392dbff6212c (DEFAULT_TABLE table=test-table [id=bd3c666a1f1c49c9917430b883568840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:33.111469 29535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 298c98786a05403895ce392dbff6212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:33.111485 29800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 298c98786a05403895ce392dbff6212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:33.115828 29668 tablet_service.cc:1467] Processing CreateTablet for tablet 298c98786a05403895ce392dbff6212c (DEFAULT_TABLE table=test-table [id=bd3c666a1f1c49c9917430b883568840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:29:33.117205 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 298c98786a05403895ce392dbff6212c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:29:33.127635 29882 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Bootstrap starting.
I20260925 10:29:33.128620 29883 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Bootstrap starting.
I20260925 10:29:33.132180 29884 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Bootstrap starting.
I20260925 10:29:33.132510 29882 tablet_bootstrap.cc:654] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:33.133121 29883 tablet_bootstrap.cc:654] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:33.133968 29882 log.cc:824] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:33.134543 29883 log.cc:824] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:33.135803 29884 tablet_bootstrap.cc:654] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Neither blocks nor log segments found. Creating new log.
I20260925 10:29:33.136989 29884 log.cc:824] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:33.137501 29882 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: No bootstrap required, opened a new log
I20260925 10:29:33.137826 29882 ts_tablet_manager.cc:1397] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 10:29:33.138880 29883 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: No bootstrap required, opened a new log
I20260925 10:29:33.139165 29883 ts_tablet_manager.cc:1397] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 10:29:33.140081 29884 tablet_bootstrap.cc:492] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: No bootstrap required, opened a new log
I20260925 10:29:33.140360 29884 ts_tablet_manager.cc:1397] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 10:29:33.146752 29883 raft_consensus.cc:348] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.147179 29883 raft_consensus.cc:374] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:33.147313 29883 raft_consensus.cc:729] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1997c44576483c89593c685c0b6d48, State: Initialized, Role: FOLLOWER
I20260925 10:29:33.148177 29883 consensus_queue.cc:260] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.148839 29882 raft_consensus.cc:348] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.149300 29882 raft_consensus.cc:374] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:33.149391 29882 raft_consensus.cc:729] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8937859fbd70412bbd3f0fe98af3daec, State: Initialized, Role: FOLLOWER
I20260925 10:29:33.150118 29882 consensus_queue.cc:260] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.151440 29865 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:33.151190 29884 raft_consensus.cc:348] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.151712 29884 raft_consensus.cc:374] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:29:33.151815 29884 raft_consensus.cc:729] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b612219a067c47f3aa592388f312fc5d, State: Initialized, Role: FOLLOWER
I20260925 10:29:33.152477 29883 ts_tablet_manager.cc:1428] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260925 10:29:33.152647 29884 consensus_queue.cc:260] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.154085 29734 heartbeater.cc:503] Master 127.12.248.126:40303 was elected leader, sending a full tablet report...
I20260925 10:29:33.156450 29884 ts_tablet_manager.cc:1428] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260925 10:29:33.158180 29882 ts_tablet_manager.cc:1428] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260925 10:29:33.171063 29888 raft_consensus.cc:482] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:29:33.171434 29888 raft_consensus.cc:504] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.173370 29888 leader_election.cc:288] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361), b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
I20260925 10:29:33.179738 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.173562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.12.248.67:44669)
0925 10:29:33.173856 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.173876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:33.174007 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:33.174450 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:33.174460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.174497 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:33.174731 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.174747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.176050 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.176054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.177912 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.177926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.179214 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.179217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.179345 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.179370 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.179443 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.179533 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 10:29:33.179878 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.173797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:44669 (local address 127.12.248.65:35361)
0925 10:29:33.174005 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.174011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.174035 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:29:33.174182 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.174190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.174294 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.174405 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:33.174903 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.175917 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.178070 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.179065 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.179222 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.179297 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.179494 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.179628 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:29:33.180836 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.173969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:53805 (local address 127.12.248.66:37617)
0925 10:29:33.174210 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.174218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.174795 (+ 577us) server_negotiation.cc:407] Connection header received
0925 10:29:33.174957 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.174964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.175044 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.175139 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:29:33.175749 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.176502 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.178641 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.179812 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.179972 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.180048 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.180459 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.180586 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:29:33.180877 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.173908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:37617 (local address 127.12.248.67:53805)
0925 10:29:33.174687 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.174704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:29:33.174829 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:29:33.175157 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:29:33.175160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:29:33.175179 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:29:33.175542 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.175557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.176682 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.176687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.178481 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:29:33.178491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.180181 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.180186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:29:33.180299 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.180321 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:29:33.180421 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:29:33.180548 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":407,"threads_started":1}
I20260925 10:29:33.180930 29555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "298c98786a05403895ce392dbff6212c" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8937859fbd70412bbd3f0fe98af3daec" is_pre_election: true
I20260925 10:29:33.181526 29555 raft_consensus.cc:2454] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 0.
I20260925 10:29:33.181763 29688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "298c98786a05403895ce392dbff6212c" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b612219a067c47f3aa592388f312fc5d" is_pre_election: true
I20260925 10:29:33.182250 29756 leader_election.cc:302] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1997c44576483c89593c685c0b6d48, 8937859fbd70412bbd3f0fe98af3daec; no voters:
I20260925 10:29:33.182257 29688 raft_consensus.cc:2454] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 0.
I20260925 10:29:33.182729 29888 raft_consensus.cc:2781] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:29:33.182891 29888 raft_consensus.cc:482] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:29:33.183043 29888 raft_consensus.cc:3037] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:33.185696 29888 raft_consensus.cc:504] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.186332 29888 leader_election.cc:288] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 election: Requested vote from peers 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361), b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617)
I20260925 10:29:33.186757 29555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "298c98786a05403895ce392dbff6212c" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8937859fbd70412bbd3f0fe98af3daec"
I20260925 10:29:33.186758 29688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "298c98786a05403895ce392dbff6212c" candidate_uuid: "0e1997c44576483c89593c685c0b6d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b612219a067c47f3aa592388f312fc5d"
I20260925 10:29:33.186947 29688 raft_consensus.cc:3037] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:33.186967 29555 raft_consensus.cc:3037] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:29:33.189332 29688 raft_consensus.cc:2454] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 1.
I20260925 10:29:33.189395 29555 raft_consensus.cc:2454] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1997c44576483c89593c685c0b6d48 in term 1.
I20260925 10:29:33.189774 29754 leader_election.cc:302] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1997c44576483c89593c685c0b6d48, b612219a067c47f3aa592388f312fc5d; no voters:
I20260925 10:29:33.190071 29888 raft_consensus.cc:2781] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:29:33.190541 29888 raft_consensus.cc:686] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1997c44576483c89593c685c0b6d48, State: Running, Role: LEADER
I20260925 10:29:33.191035 29888 consensus_queue.cc:237] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } }
I20260925 10:29:33.195845 29414 catalog_manager.cc:5495] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e1997c44576483c89593c685c0b6d48 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "0e1997c44576483c89593c685c0b6d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e1997c44576483c89593c685c0b6d48" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 36031 } health_report { overall_health: HEALTHY } } }
I20260925 10:29:33.221575 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.216487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.12.248.67:36031)
0925 10:29:33.216688 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.216693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.216712 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:29:33.216905 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.216909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.217000 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.217135 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:29:33.217520 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.218225 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.219522 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.219902 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.219991 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.220463 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:29:33.220968 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:29:33.221141 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.221252 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.221318 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:29:33.229563 29688 raft_consensus.cc:1261] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Refusing update from remote peer 0e1997c44576483c89593c685c0b6d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:29:33.229669 29555 raft_consensus.cc:1261] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Refusing update from remote peer 0e1997c44576483c89593c685c0b6d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:29:33.230585 29892 consensus_queue.cc:1035] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b612219a067c47f3aa592388f312fc5d" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:29:33.230929 29888 consensus_queue.cc:1035] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8937859fbd70412bbd3f0fe98af3daec" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:29:33.243108 29894 mvcc.cc:205] Tried to move back new op lower bound from 7333200581538897920 to 7333200581394628608. Current Snapshot: MvccSnapshot[applied={T|T < 7333200581538897920}]
I20260925 10:29:33.258572 29415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:29:33.258852 29415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:29:33.262270 29415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Sending DeleteTablet for 3 replicas of tablet 298c98786a05403895ce392dbff6212c
I20260925 10:29:33.263686 29668 tablet_service.cc:1514] Processing DeleteTablet for tablet 298c98786a05403895ce392dbff6212c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:39574
I20260925 10:29:33.263686 29535 tablet_service.cc:1514] Processing DeleteTablet for tablet 298c98786a05403895ce392dbff6212c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46516
I20260925 10:29:33.265017 29800 tablet_service.cc:1514] Processing DeleteTablet for tablet 298c98786a05403895ce392dbff6212c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:29:33 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:56498
I20260925 10:29:33.265969 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29452
I20260925 10:29:33.274653 29905 tablet_replica.cc:330] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: stopping tablet replica
I20260925 10:29:33.274909 29906 tablet_replica.cc:330] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: stopping tablet replica
I20260925 10:29:33.275650 29906 raft_consensus.cc:2229] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:29:33.276232 29905 raft_consensus.cc:2229] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:29:33.276551 29906 raft_consensus.cc:2258] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:33.277370 29904 tablet_replica.cc:330] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: stopping tablet replica
I20260925 10:29:33.277878 29904 raft_consensus.cc:2229] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:29:33.278487 29904 raft_consensus.cc:2258] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:33.279253 29906 ts_tablet_manager.cc:1905] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:33.279526 29905 raft_consensus.cc:2258] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:29:33.282150 29905 ts_tablet_manager.cc:1905] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:33.286484 29904 ts_tablet_manager.cc:1905] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:33.287132 29906 ts_tablet_manager.cc:1918] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:33.287360 29906 log.cc:1196] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/298c98786a05403895ce392dbff6212c
I20260925 10:29:33.288033 29906 ts_tablet_manager.cc:1939] T 298c98786a05403895ce392dbff6212c P 0e1997c44576483c89593c685c0b6d48: Deleting consensus metadata
I20260925 10:29:33.290089 29399 catalog_manager.cc:4841] TS 0e1997c44576483c89593c685c0b6d48 (127.12.248.67:36031): tablet 298c98786a05403895ce392dbff6212c (table test-table [id=bd3c666a1f1c49c9917430b883568840]) successfully deleted
I20260925 10:29:33.291458 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.267001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.12.248.65:35361)
0925 10:29:33.267520 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.267527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.267547 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:29:33.267608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.267614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.267707 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.267811 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.270187 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.271612 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.274110 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.274891 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.275036 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.281077 (+ 6041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.281099 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.281119 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.281170 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.285621 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.288481 (+ 2860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.288491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.288496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.288663 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.289562 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.289568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.289572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.290108 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.290465 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.290760 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.291011 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260925 10:29:33.292241 29905 ts_tablet_manager.cc:1918] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:33.292353 29595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:33.292478 29905 log.cc:1196] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/298c98786a05403895ce392dbff6212c
I20260925 10:29:33.293186 29905 ts_tablet_manager.cc:1939] T 298c98786a05403895ce392dbff6212c P b612219a067c47f3aa592388f312fc5d: Deleting consensus metadata
I20260925 10:29:33.295542 29400 catalog_manager.cc:4841] TS b612219a067c47f3aa592388f312fc5d (127.12.248.66:37617): tablet 298c98786a05403895ce392dbff6212c (table test-table [id=bd3c666a1f1c49c9917430b883568840]) successfully deleted
I20260925 10:29:33.412698 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29452
W20260925 10:29:33.430637 29402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.248.65:35361 (error 108)
I20260925 10:29:33.430754 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29604
I20260925 10:29:33.433066 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.432440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41830)
0925 10:29:33.432738 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.432866 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260925 10:29:33.433343 29402 catalog_manager.cc:4584] TS 8937859fbd70412bbd3f0fe98af3daec (127.12.248.65:35361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 298c98786a05403895ce392dbff6212c: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
I20260925 10:29:33.451831 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.431674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.12.248.66:37617)
0925 10:29:33.432195 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.432203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.432228 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:29:33.432302 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.432306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.432369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.432497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.434045 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.434796 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.436138 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.436547 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.436690 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.444204 (+ 7514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.444226 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.444248 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.444317 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.448609 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.449294 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.449309 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.449312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.449609 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.449975 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.449982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.449986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.450560 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.450883 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.451170 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.451425 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260925 10:29:33.452701 29729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:33.487427 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.486835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41832)
0925 10:29:33.487098 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.487218 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 10:29:33.488719 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.488269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41838)
0925 10:29:33.488459 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.488568 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:29:33.549511 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.548834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41854)
0925 10:29:33.549134 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.549261 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 10:29:33.550751 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.550333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41870)
0925 10:29:33.550505 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.550602 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:29:33.555967 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29604
I20260925 10:29:33.574244 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29737
I20260925 10:29:33.592135 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.575203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.12.248.67:36031)
0925 10:29:33.575512 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 10:29:33.575519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:29:33.575543 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:29:33.575617 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:29:33.575624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:29:33.575719 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:29:33.575884 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:29:33.577271 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.578707 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.580114 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:29:33.580908 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:29:33.581059 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:29:33.584006 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:29:33.584022 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:29:33.584044 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:29:33.584090 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:29:33.588779 (+ 4689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.589660 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.589671 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.589675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.589878 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:29:33.590293 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:29:33.590300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:29:33.590305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:29:33.590872 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:29:33.591250 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 10:29:33.591435 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:29:33.591683 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260925 10:29:33.593091 29860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:33.631278 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.630758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41878)
0925 10:29:33.630988 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.631110 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 10:29:33.632589 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:33.632046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35361 (local address 127.0.0.1:41880)
0925 10:29:33.632355 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:33.632431 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260925 10:29:33.716382 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29737
I20260925 10:29:33.735924 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29382
I20260925 10:29:33.737140 29445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:29:33.854916 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29382
I20260925 10:29:53.877635 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--webserver_interface=127.12.248.126
--webserver_port=35751
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:54.223124 29912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:54.223474 29912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:54.223656 29912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:54.231668 29912 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:29:54.231806 29912 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:29:54.231873 29912 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:29:54.231943 29912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:29:54.232018 29912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:54.232112 29912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:54.232203 29912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:29:54.232283 29912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:29:54.242363 29912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=35751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:54.243767 29912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:54.245151 29912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:54.257112 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:54.257215 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:54.257182 29920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:54.258610 29912 server_base.cc:1034] running on GCE node
I20260925 10:29:54.259428 29912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:54.260921 29912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:54.262188 29912 hybrid_clock.cc:648] HybridClock initialized: now 1790332194262116 us; error 46 us; skew 500 ppm
I20260925 10:29:54.262781 29912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:54 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:54.275761 29912 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:29:54.277283 29912 webserver.cc:460] Webserver started at http://127.12.248.126:35751/ using document root <none> and password file <none>
I20260925 10:29:54.278056 29912 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:54.278149 29912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:54.283850 29912 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 10:29:54.286753 29927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:54.288168 29912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:29:54.288314 29912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "794dd97b6f5842669806dffedf61a5f8"
format_stamp: "Formatted at 2026-09-25 10:29:08 on dist-test-slave-f9rl"
server_key: "097d7ed0bca0129c05e298e3dd83c08d"
server_key_iv: "04c87bd4ed3e8e72503dc9b6f338ef3d"
server_key_version: "encryptionkey@0"
I20260925 10:29:54.289403 29912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:54.338861 29912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:54.343258 29912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:54.343566 29912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:54.363104 29912 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:40303
I20260925 10:29:54.363108 29978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:40303 every 8 connection(s)
I20260925 10:29:54.364790 29912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:29:54.371625 29979 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:29:54.374449 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29912
I20260925 10:29:54.375072 29979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap starting.
I20260925 10:29:54.375440 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:35361
--local_ip_for_outbound_sockets=127.12.248.65
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=36853
--webserver_interface=127.12.248.65
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:29:54.401202 29979 log.cc:824] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:29:54.422717 29979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:29:54.423403 29979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Bootstrap complete.
I20260925 10:29:54.437161 29979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:54.438531 29979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Initialized, Role: FOLLOWER
I20260925 10:29:54.439283 29979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:54.439571 29979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:29:54.439707 29979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:29:54.439882 29979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:29:54.446307 29979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:54.446925 29979 leader_election.cc:302] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794dd97b6f5842669806dffedf61a5f8; no voters:
I20260925 10:29:54.447608 29979 leader_election.cc:288] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:29:54.447819 29983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:29:54.449018 29983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [term 3 LEADER]: Becoming Leader. State: Replica: 794dd97b6f5842669806dffedf61a5f8, State: Running, Role: LEADER
I20260925 10:29:54.449626 29983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } }
I20260925 10:29:54.449920 29979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:29:54.455612 29984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:54.455974 29984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:54.457127 29985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794dd97b6f5842669806dffedf61a5f8. Latest consensus state: current_term: 3 leader_uuid: "794dd97b6f5842669806dffedf61a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794dd97b6f5842669806dffedf61a5f8" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40303 } } }
I20260925 10:29:54.457860 29985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:29:54.464010 29988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:29:54.472558 29988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:29:54.474468 29988 catalog_manager.cc:1259] Loaded cluster ID: bb828e79c7cc46699c3417cc4a93ed7b
I20260925 10:29:54.474629 29988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:29:54.480038 29988 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:29:54.482180 29988 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Loaded TSK: 0
I20260925 10:29:54.482359 29988 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 794dd97b6f5842669806dffedf61a5f8: Loaded TSK: 1
F20260925 10:29:54.483654 29988 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5a2f5b49ad google::LogMessage::Fail() at ??:0
@ 0x7f5a2f5b788a google::LogMessage::SendToLog() at ??:0
@ 0x7f5a2f5b44de google::LogMessage::Flush() at ??:0
@ 0x7f5a2f5b5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5a333187d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5a3330773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5a44fe31c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5a44fe2783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5a4506e2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5a4506e0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5a450b85e9 std::function<>::operator()() at ??:0
@ 0x7f5a44fe6cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5a44fe4c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5a4506c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5a4506c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a32200345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5a3220391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5a32203799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5a321deeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5a354b86db start_thread at ??:0
@ 0x7f5a2e44e61f clone at ??:0
W20260925 10:29:54.772367 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:54.772778 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:54.772960 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:54.780375 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:54.780524 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:54.780712 29981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:29:54.790993 29981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=36853
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:54.792392 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:54.793852 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:54.807391 30027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:54.807467 30030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:54.807422 30028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:54.808249 29981 server_base.cc:1034] running on GCE node
I20260925 10:29:54.808869 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:54.809926 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:54.811182 29981 hybrid_clock.cc:648] HybridClock initialized: now 1790332194811125 us; error 48 us; skew 500 ppm
I20260925 10:29:54.811542 29981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:54 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332194, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:54.815011 29981 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:29:54.816555 29981 webserver.cc:460] Webserver started at http://127.12.248.65:36853/ using document root <none> and password file <none>
I20260925 10:29:54.817265 29981 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:54.817412 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:54.823004 29981 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 10:29:54.826297 30037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:54.827777 29981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 10:29:54.827976 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "8937859fbd70412bbd3f0fe98af3daec"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "59211fccb3015c6d063dd8fcd518af9a"
server_key_iv: "7bf48736cd7b39ab8bd63a6a92143bd2"
server_key_version: "encryptionkey@0"
I20260925 10:29:54.829038 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:54.849016 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:54.853484 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:54.853799 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:54.854853 29981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:54.856637 30044 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 10:29:54.862066 30044 ts_tablet_manager.cc:1852] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 10:29:54.863210 30044 ts_tablet_manager.cc:1905] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:29:54.871016 30044 ts_tablet_manager.cc:1918] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:29:54.871289 30044 log.cc:1196] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/298c98786a05403895ce392dbff6212c
I20260925 10:29:54.871797 30044 ts_tablet_manager.cc:1939] T 298c98786a05403895ce392dbff6212c P 8937859fbd70412bbd3f0fe98af3daec: Deleting consensus metadata
I20260925 10:29:54.872223 29981 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 10:29:54.872345 29981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260925 10:29:54.872448 29981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:54.872506 29981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:54.909205 29981 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:35361
I20260925 10:29:54.909312 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:35361 every 8 connection(s)
I20260925 10:29:54.910619 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:29:54.915618 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 29981
I20260925 10:29:54.916143 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:54.914094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:37301)
0925 10:29:54.914873 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:54.915162 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":292,"threads_started":1}
I20260925 10:29:54.916433 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:37617
--local_ip_for_outbound_sockets=127.12.248.66
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=36137
--webserver_interface=127.12.248.66
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:29:54.916796 30151 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:40303 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:40303: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
I20260925 10:29:54.917695 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:54.917335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:54879)
0925 10:29:54.917517 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:54.917596 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:29:54.918515 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:54.918184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:59047)
0925 10:29:54.918321 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:54.918404 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:29:54.918705 30151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:29:55.264215 30154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:55.264536 30154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:55.264640 30154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:55.272291 30154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:55.272416 30154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:55.272557 30154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:29:55.282806 30154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:37617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=36137
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:55.284117 30154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:55.285496 30154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:55.299535 30159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:55.299688 30162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:55.299706 30160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:55.300892 30154 server_base.cc:1034] running on GCE node
I20260925 10:29:55.301597 30154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:55.302600 30154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:55.303836 30154 hybrid_clock.cc:648] HybridClock initialized: now 1790332195303769 us; error 58 us; skew 500 ppm
I20260925 10:29:55.304162 30154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:55 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332195, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:55.307482 30154 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:29:55.309058 30154 webserver.cc:460] Webserver started at http://127.12.248.66:36137/ using document root <none> and password file <none>
I20260925 10:29:55.309851 30154 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:55.310005 30154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:55.315574 30154 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 10:29:55.318709 30169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.320127 30154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:29:55.320340 30154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "b612219a067c47f3aa592388f312fc5d"
format_stamp: "Formatted at 2026-09-25 10:29:09 on dist-test-slave-f9rl"
server_key: "c44d0e76ee58e43c3fc3051955113f2a"
server_key_iv: "64e4e4c463831c90a51c6b7e751326b9"
server_key_version: "encryptionkey@0"
I20260925 10:29:55.321401 30154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:55.342597 30154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:55.347472 30154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:55.347767 30154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:55.348802 30154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:55.350224 30154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:55.350358 30154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.350502 30154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:55.350591 30154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.385938 30154 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:37617
I20260925 10:29:55.386030 30281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:37617 every 8 connection(s)
I20260925 10:29:55.387519 30154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:29:55.392622 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.390597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:40463)
0925 10:29:55.391372 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.391713 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":317,"threads_started":1}
W20260925 10:29:55.393296 30282 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:40303 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:40303: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
I20260925 10:29:55.394256 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30154
I20260925 10:29:55.394336 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.393898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:52341)
0925 10:29:55.394096 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.394188 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:29:55.394987 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:36031
--local_ip_for_outbound_sockets=127.12.248.67
--tserver_master_addrs=127.12.248.126:40303
--webserver_port=35779
--webserver_interface=127.12.248.67
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:29:55.395426 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.394983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:45701)
0925 10:29:55.395157 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.395263 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 10:29:55.395742 30282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:29:55.723331 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:29:55.723624 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:29:55.723822 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:29:55.731845 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:29:55.731964 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:29:55.732198 30285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:29:55.742465 30285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:36031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=35779
--tserver_master_addrs=127.12.248.126:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:29:55.743916 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:29:55.745290 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:29:55.758899 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:55.758937 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:29:55.758957 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:29:55.760239 30285 server_base.cc:1034] running on GCE node
I20260925 10:29:55.760840 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:29:55.761835 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:29:55.763089 30285 hybrid_clock.cc:648] HybridClock initialized: now 1790332195763040 us; error 28 us; skew 500 ppm
I20260925 10:29:55.763427 30285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:29:55 dist-test-slave-f9rl krb5kdc[28826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332195, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:29:55.766780 30285 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:29:55.768332 30285 webserver.cc:460] Webserver started at http://127.12.248.67:35779/ using document root <none> and password file <none>
I20260925 10:29:55.769029 30285 fs_manager.cc:359] Metadata directory not provided
I20260925 10:29:55.769179 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:29:55.774946 30285 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:29:55.779654 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.781107 30285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:29:55.781304 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "0e1997c44576483c89593c685c0b6d48"
format_stamp: "Formatted at 2026-09-25 10:29:10 on dist-test-slave-f9rl"
server_key: "f4bb3719f3537827a93c14851832115b"
server_key_iv: "5950af662a475b88e07d5e8a64115ff4"
server_key_version: "encryptionkey@0"
I20260925 10:29:55.782397 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:29:55.801046 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:29:55.805879 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:29:55.806171 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:29:55.807188 30285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:29:55.808511 30285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:29:55.808647 30285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.808775 30285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:29:55.808867 30285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:29:55.844758 30285 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:36031
I20260925 10:29:55.844810 30412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:36031 every 8 connection(s)
I20260925 10:29:55.846618 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:29:55.849767 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30285
I20260925 10:29:55.852438 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.849881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:60149)
0925 10:29:55.850941 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.851233 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":361,"threads_started":1}
W20260925 10:29:55.853152 30413 heartbeater.cc:644] Failed to heartbeat to 127.12.248.126:40303 (0 consecutive failures): Network error: Failed to ping master at 127.12.248.126:40303: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
I20260925 10:29:55.854189 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.853731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:49885)
0925 10:29:55.853933 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.854028 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:29:55.855167 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.854832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:40937)
0925 10:29:55.854985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.855061 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 10:29:55.855391 30413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:29:55.920220 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:55.919337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:42621)
0925 10:29:55.919880 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:55.919969 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
I20260925 10:29:56.397285 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:56.396363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:33921)
0925 10:29:56.396931 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:56.397044 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":332,"threads_started":1}
I20260925 10:29:56.857089 30419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:56.855965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:54335)
0925 10:29:56.856616 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:56.856767 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":305,"threads_started":1}
I20260925 10:29:56.922011 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:56.921218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:55921)
0925 10:29:56.921757 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:56.921842 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":315,"threads_started":1}
I20260925 10:29:57.399098 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:57.398297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:45277)
0925 10:29:57.398809 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:57.398896 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":301,"threads_started":1}
I20260925 10:29:57.859153 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:57.858353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:50583)
0925 10:29:57.858897 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:57.858981 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
I20260925 10:29:57.924319 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:57.923275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:56037)
0925 10:29:57.923910 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:57.924027 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":296,"threads_started":1}
I20260925 10:29:58.400894 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:58.400063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:53113)
0925 10:29:58.400625 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:58.400713 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
I20260925 10:29:58.861097 30425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:58.860234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:55211)
0925 10:29:58.860816 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:58.860897 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":369,"threads_started":1}
I20260925 10:29:58.926165 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:58.925337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:44529)
0925 10:29:58.925892 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:58.925996 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":347,"threads_started":1}
I20260925 10:29:59.402696 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:59.401943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:51069)
0925 10:29:59.402378 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:59.402459 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260925 10:29:59.862794 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:59.862055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:52439)
0925 10:29:59.862492 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:59.862572 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260925 10:29:59.928260 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:29:59.927432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:49105)
0925 10:29:59.927989 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 10:29:59.928075 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":301,"threads_started":1}
I20260925 10:30:00.404412 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:00.403667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:59331)
0925 10:30:00.404116 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:00.404243 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260925 10:30:00.864614 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:00.863834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:54251)
0925 10:30:00.864318 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:00.864419 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260925 10:30:00.930105 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:00.929247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:35349)
0925 10:30:00.929815 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:00.929921 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":334,"threads_started":1}
I20260925 10:30:01.406212 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:01.405360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:41855)
0925 10:30:01.405918 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:01.406018 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":315,"threads_started":1}
I20260925 10:30:01.866417 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:01.865691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:52229)
0925 10:30:01.866150 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:01.866249 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260925 10:30:01.932015 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:01.931154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:48683)
0925 10:30:01.931711 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:01.931800 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":240,"threads_started":1}
I20260925 10:30:02.408138 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:02.407369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:42033)
0925 10:30:02.407845 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:02.407922 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":224,"threads_started":1}
I20260925 10:30:02.868206 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:02.867351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:37139)
0925 10:30:02.867902 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:02.867981 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
I20260925 10:30:02.934208 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:02.933107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:36829)
0925 10:30:02.933916 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:02.934032 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":302,"threads_started":1}
I20260925 10:30:03.410318 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:03.409338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:49033)
0925 10:30:03.410037 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:03.410127 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":433,"threads_started":1}
I20260925 10:30:03.870082 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:03.869230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:46315)
0925 10:30:03.869825 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:03.869915 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":332,"threads_started":1}
I20260925 10:30:03.936116 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:03.935270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:39037)
0925 10:30:03.935813 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:03.935923 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":241,"threads_started":1}
I20260925 10:30:04.412323 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:04.411474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:33185)
0925 10:30:04.412019 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:04.412132 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":281,"threads_started":1}
I20260925 10:30:04.871984 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:04.871195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:34911)
0925 10:30:04.871715 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:04.871805 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
I20260925 10:30:04.938264 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:04.937332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:60937)
0925 10:30:04.937952 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:04.938067 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":340,"threads_started":1}
I20260925 10:30:05.414220 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:05.413418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:54971)
0925 10:30:05.413960 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:05.414051 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":246,"threads_started":1}
I20260925 10:30:05.873863 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:05.873091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:44385)
0925 10:30:05.873593 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:05.873687 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":241,"threads_started":1}
I20260925 10:30:05.940618 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:05.939373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:38567)
0925 10:30:05.940149 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:05.940299 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":425,"threads_started":1}
I20260925 10:30:06.415946 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:06.415175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:51085)
0925 10:30:06.415662 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:06.415761 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260925 10:30:06.875715 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:06.874876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:53335)
0925 10:30:06.875399 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:06.875515 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":280,"threads_started":1}
I20260925 10:30:06.942502 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:06.941726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:59935)
0925 10:30:06.942181 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:06.942270 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260925 10:30:07.417970 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:07.417030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:35899)
0925 10:30:07.417696 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:07.417790 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":288,"threads_started":1}
I20260925 10:30:07.877765 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:07.876857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:59469)
0925 10:30:07.877412 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:07.877560 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":312,"threads_started":1}
I20260925 10:30:07.944532 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:07.943716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:52361)
0925 10:30:07.944227 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:07.944337 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":242,"threads_started":1}
I20260925 10:30:08.419884 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:08.418996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:47555)
0925 10:30:08.419593 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:08.419697 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":282,"threads_started":1}
I20260925 10:30:08.879778 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:08.878894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:46317)
0925 10:30:08.879436 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:08.879545 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":245,"threads_started":1}
I20260925 10:30:08.946465 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:08.945688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:46933)
0925 10:30:08.946169 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:08.946273 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":236,"threads_started":1}
I20260925 10:30:09.421694 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:09.420825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:45303)
0925 10:30:09.421351 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:09.421489 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":322,"threads_started":1}
I20260925 10:30:09.881982 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:09.880977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:39851)
0925 10:30:09.881697 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:09.881788 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":463,"threads_started":1}
I20260925 10:30:09.948791 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:09.947951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.65:52947)
0925 10:30:09.948455 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:09.948580 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260925 10:30:10.423642 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:10.422880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.66:52631)
0925 10:30:10.423347 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:10.423445 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":248,"threads_started":1}
I20260925 10:30:10.859877 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 29981
I20260925 10:30:10.883961 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:10.883153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:40303 (local address 127.12.248.67:43991)
0925 10:30:10.883673 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:10.883776 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.126:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":277,"threads_started":1}
I20260925 10:30:10.887476 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:10.861203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.12.248.65:35361)
0925 10:30:10.861856 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:10.861871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:10.861908 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:30:10.863648 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:10.863658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:10.864121 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:10.864506 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:10.866165 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:10.867458 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:10.868789 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:10.869550 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:10.869755 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:10.880137 (+ 10382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:10.880146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:10.880165 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:10.880212 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:10.883240 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:10.884045 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:10.884052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:10.884060 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:10.884311 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:10.884850 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:10.884858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:10.884863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:10.885748 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:10.886000 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:10.886644 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 10:30:10.887094 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":318,"threads_started":1}
I20260925 10:30:10.889045 30146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:10.978212 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 29981
I20260925 10:30:10.995957 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30154
I20260925 10:30:11.014756 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:10.997095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.12.248.66:37617)
0925 10:30:10.997741 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:10.997755 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:10.997795 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:30:10.999442 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:10.999452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:10.999895 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:11.000207 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:11.001818 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:11.003061 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:11.004482 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:11.005158 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:11.005294 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:11.008375 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:11.008383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:11.008398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:11.008434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:11.011399 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:11.012182 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:11.012188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:11.012195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:11.012362 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:11.012779 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:11.012782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:11.012784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:11.013336 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:11.013657 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:11.014101 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 10:30:11.014467 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":342,"threads_started":1}
I20260925 10:30:11.016222 30277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:11.106369 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30154
I20260925 10:30:11.124811 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30285
I20260925 10:30:11.143252 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:11.126076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.12.248.67:36031)
0925 10:30:11.126682 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:11.126697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:11.126738 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:30:11.128400 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:11.128409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:11.128873 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:11.129176 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:11.130639 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:11.132031 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:11.133380 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:11.134137 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:11.134285 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:11.137004 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:11.137011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:11.137026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:11.137074 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:11.140014 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:11.140712 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:11.140719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:11.140726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:11.140886 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:11.141281 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:11.141284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:11.141286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:11.141905 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:11.142161 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:11.142579 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 10:30:11.142918 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":275,"threads_started":1}
I20260925 10:30:11.144702 30408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:11.228935 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30285
2026-09-25T10:30:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64604 ms)
[----------] 30 tests from SecurityITest (304954 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:11 dist-test-slave-f9rl krb5kdc[30471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:11 dist-test-slave-f9rl krb5kdc[30471](info): set up 2 sockets
Sep 25 10:30:11 dist-test-slave-f9rl krb5kdc[30471](info): commencing operation
krb5kdc: starting...
W20260925 10:30:12.355744 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:12 dist-test-slave-f9rl krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:30:12.614161 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:39551
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:39551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:12.948521 30484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:12.948892 30484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:12.949065 30484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:12.956504 30484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:12.956676 30484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:12.956779 30484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:12.956885 30484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:12.956974 30484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:12.968582 30484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:39551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:12.970122 30484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:12.971527 30484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:12.984239 30492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:12.984308 30489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:12.984277 30490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:12.984513 30484 server_base.cc:1034] running on GCE node
I20260925 10:30:12.985885 30484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:12.987330 30484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:12.988533 30484 hybrid_clock.cc:648] HybridClock initialized: now 1790332212988441 us; error 73 us; skew 500 ppm
I20260925 10:30:12.988893 30484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:12 dist-test-slave-f9rl krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:12.992089 30484 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:30:12.993808 30484 webserver.cc:460] Webserver started at http://127.12.248.126:44877/ using document root <none> and password file <none>
I20260925 10:30:12.994529 30484 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:12.994637 30484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:12.994961 30484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:12.998139 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5"
format_stamp: "Formatted at 2026-09-25 10:30:12 on dist-test-slave-f9rl"
server_key: "bb2b43b8a72f093bab60072585b7eff0"
server_key_iv: "bb194973a7dca3bbda558ab85d09f4c3"
server_key_version: "encryptionkey@0"
I20260925 10:30:12.998912 30484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5"
format_stamp: "Formatted at 2026-09-25 10:30:12 on dist-test-slave-f9rl"
server_key: "bb2b43b8a72f093bab60072585b7eff0"
server_key_iv: "bb194973a7dca3bbda558ab85d09f4c3"
server_key_version: "encryptionkey@0"
I20260925 10:30:13.004242 30484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:13.008071 30499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:13.009622 30484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:30:13.009830 30484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5"
format_stamp: "Formatted at 2026-09-25 10:30:12 on dist-test-slave-f9rl"
server_key: "bb2b43b8a72f093bab60072585b7eff0"
server_key_iv: "bb194973a7dca3bbda558ab85d09f4c3"
server_key_version: "encryptionkey@0"
I20260925 10:30:13.010061 30484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:13.024881 30484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:13.029541 30484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:13.029873 30484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:13.050576 30484 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:39551
I20260925 10:30:13.050582 30551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:39551 every 8 connection(s)
I20260925 10:30:13.052137 30484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:13.055866 30552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:13.059425 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30484
I20260925 10:30:13.059731 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:13.060461 13281 external_mini_cluster.cc:1428] Setting key 910169928d052311814a2d0faf9dc5da
I20260925 10:30:13.066896 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: Bootstrap starting.
I20260925 10:30:13.070832 30552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:13.072170 30552 log.cc:824] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:13.075740 30552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: No bootstrap required, opened a new log
Sep 25 10:30:13 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:13.083843 30552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } }
I20260925 10:30:13.084257 30552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:13.084376 30552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7f9b473dcb4c66a2bb98c45eae40c5, State: Initialized, Role: FOLLOWER
I20260925 10:30:13.085083 30552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } }
I20260925 10:30:13.085386 30552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:13.085592 30552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:13.085799 30552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:13.087021 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:13.062627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.12.248.126:39551)
0925 10:30:13.063497 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:13.063513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:13.063558 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:30:13.065650 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:13.065675 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:13.066093 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:13.066450 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:13.068366 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.069936 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.072121 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.073115 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.073332 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:13.078635 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:13.078643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:13.078661 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:13.078700 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:13.082291 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:13.083125 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:13.083133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:13.083140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:13.083323 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:13.083750 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:13.083757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:13.083760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:13.084440 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:13.084784 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:13.085264 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 10:30:13.085842 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":406,"threads_started":1}
I20260925 10:30:13.088608 30552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } }
I20260925 10:30:13.089264 30552 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7f9b473dcb4c66a2bb98c45eae40c5; no voters:
I20260925 10:30:13.098760 30552 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:13.098886 30557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:13.099858 30557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7f9b473dcb4c66a2bb98c45eae40c5, State: Running, Role: LEADER
I20260925 10:30:13.100484 30557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } }
I20260925 10:30:13.101619 30552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:13.104212 30558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7f9b473dcb4c66a2bb98c45eae40c5. Latest consensus state: current_term: 1 leader_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } } }
I20260925 10:30:13.104789 30558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:13.104595 30559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7f9b473dcb4c66a2bb98c45eae40c5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 39551 } } }
I20260925 10:30:13.105732 30559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:13.108848 30565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:13.115197 30565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:13.124939 30565 catalog_manager.cc:1347] Generated new cluster ID: efd543dbc97743acb11df7c107cfee9b
I20260925 10:30:13.125150 30565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:13.168470 30565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:13.170470 30565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:13.183621 30565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: Generated new TSK 0
I20260925 10:30:13.184715 30565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:30:13.254266 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:13.594311 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:13.594594 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:13.594753 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:13.602109 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:13.602243 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:13.602447 30578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:13.612525 30578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:13.613858 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:13.615304 30578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:13.628614 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:13.628686 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:13.628729 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:13.629859 30578 server_base.cc:1034] running on GCE node
I20260925 10:30:13.630543 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:13.631608 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:13.632881 30578 hybrid_clock.cc:648] HybridClock initialized: now 1790332213632811 us; error 54 us; skew 500 ppm
I20260925 10:30:13.633256 30578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:13 dist-test-slave-f9rl krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:13.636863 30578 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:30:13.638475 30578 webserver.cc:460] Webserver started at http://127.12.248.65:43613/ using document root <none> and password file <none>
I20260925 10:30:13.639173 30578 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:13.639287 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:13.639606 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:13.642875 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "56015e9a11394c9f9b3f7ec37802a616"
format_stamp: "Formatted at 2026-09-25 10:30:13 on dist-test-slave-f9rl"
server_key: "8dd00360b33fec5ba1c05e5b24e5d9e4"
server_key_iv: "5cb6379b00fea006582201b5c933ce90"
server_key_version: "encryptionkey@0"
I20260925 10:30:13.643672 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "56015e9a11394c9f9b3f7ec37802a616"
format_stamp: "Formatted at 2026-09-25 10:30:13 on dist-test-slave-f9rl"
server_key: "8dd00360b33fec5ba1c05e5b24e5d9e4"
server_key_iv: "5cb6379b00fea006582201b5c933ce90"
server_key_version: "encryptionkey@0"
I20260925 10:30:13.649343 30578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:30:13.653364 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:13.654932 30578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:30:13.655072 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "56015e9a11394c9f9b3f7ec37802a616"
format_stamp: "Formatted at 2026-09-25 10:30:13 on dist-test-slave-f9rl"
server_key: "8dd00360b33fec5ba1c05e5b24e5d9e4"
server_key_iv: "5cb6379b00fea006582201b5c933ce90"
server_key_version: "encryptionkey@0"
I20260925 10:30:13.655259 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:13.668999 30578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:13.673318 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:13.673635 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:13.674556 30578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:13.675840 30578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:13.675913 30578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:13.675995 30578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:13.676043 30578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:13.709245 30578 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:44069
I20260925 10:30:13.709323 30706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:44069 every 8 connection(s)
I20260925 10:30:13.710770 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:13.712678 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30578
I20260925 10:30:13.712913 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:13.713821 13281 external_mini_cluster.cc:1428] Setting key a7fa294a9915c6718bea74710ecff3ce
Sep 25 10:30:13 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332213, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:13.732937 30709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:13.713557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:47969 (local address 127.12.248.126:39551)
0925 10:30:13.714023 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:13.714028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:13.715388 (+ 1360us) server_negotiation.cc:407] Connection header received
0925 10:30:13.717468 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:13.717474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:13.717555 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:13.717683 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:13.719212 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.719927 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.721563 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.721985 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.722131 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:13.726585 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:13.726590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:13.726593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:13.726629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:13.729296 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:13.730393 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:13.730398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:13.730400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:13.730524 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:13.731082 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:13.731086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:13.731088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:13.731370 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:13.731484 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:13.732476 (+ 992us) server_negotiation.cc:299] Negotiation successful
0925 10:30:13.732671 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260925 10:30:13.734678 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:13.714234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:39551 (local address 127.12.248.65:47969)
0925 10:30:13.715209 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:13.715265 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:13.717163 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:13.717936 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:13.717945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:13.718463 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:13.719038 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:13.719056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.720096 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.720102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:13.721362 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:13.721375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:13.722222 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:13.722231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:13.722483 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:13.723165 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:13.723198 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:13.726315 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:13.729523 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:13.729536 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:13.729559 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:13.730258 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:13.730659 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:13.730667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:13.730674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:13.730953 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:13.731509 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:13.731520 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:13.732047 (+ 527us) client_negotiation.cc:769] Sending connection context
0925 10:30:13.732365 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 10:30:13.733004 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":448,"threads_started":1}
I20260925 10:30:13.736500 30707 heartbeater.cc:347] Connected to a master server at 127.12.248.126:39551
I20260925 10:30:13.736889 30707 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:13.737913 30707 heartbeater.cc:511] Master 127.12.248.126:39551 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
I20260925 10:30:13.740520 30516 ts_manager.cc:194] Registered new tserver with Master: 56015e9a11394c9f9b3f7ec37802a616 (127.12.248.65:44069)
I20260925 10:30:13.742076 30516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:47969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:30:13.774900 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:14.112473 30713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:14.112787 30713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:14.112974 30713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:14.120734 30713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:14.120853 30713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:14.121052 30713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:14.131326 30713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:14.132664 30713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:14.134217 30713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:14.147887 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:14.148018 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:14.148536 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:14.148900 30713 server_base.cc:1034] running on GCE node
I20260925 10:30:14.149624 30713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:14.150604 30713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:14.151852 30713 hybrid_clock.cc:648] HybridClock initialized: now 1790332214151815 us; error 46 us; skew 500 ppm
I20260925 10:30:14.152197 30713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:14 dist-test-slave-f9rl krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332214, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:14.155382 30713 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:30:14.156874 30713 webserver.cc:460] Webserver started at http://127.12.248.66:41589/ using document root <none> and password file <none>
I20260925 10:30:14.157593 30713 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:14.157702 30713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:14.157967 30713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:14.161031 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "b7ebe4fba19241a3b098706c2040126f"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "010099551b3e074a19276d0d87ed3888"
server_key_iv: "d0334af1533cc48f3799d83837573bac"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.161846 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ebe4fba19241a3b098706c2040126f"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "010099551b3e074a19276d0d87ed3888"
server_key_iv: "d0334af1533cc48f3799d83837573bac"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.167016 30713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:30:14.171108 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.173117 30713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 10:30:14.173326 30713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "b7ebe4fba19241a3b098706c2040126f"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "010099551b3e074a19276d0d87ed3888"
server_key_iv: "d0334af1533cc48f3799d83837573bac"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.173604 30713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:14.200048 30713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:14.205060 30713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:14.205382 30713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:14.206480 30713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:14.207881 30713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:14.207985 30713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.208141 30713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:14.208220 30713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.242261 30713 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:33169
I20260925 10:30:14.242354 30841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:33169 every 8 connection(s)
I20260925 10:30:14.243651 30713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:14.250272 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30713
I20260925 10:30:14.250489 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:14.250954 13281 external_mini_cluster.cc:1428] Setting key 2b2ab37f31142d60330d4727adc712a2
Sep 25 10:30:14 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332214, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:14.267555 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.246448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:38645 (local address 127.12.248.126:39551)
0925 10:30:14.246928 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.246932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.247963 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 10:30:14.249995 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.250000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.250104 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.250225 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:14.252089 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.252951 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.255720 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.256172 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.256285 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.260843 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:14.260853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:14.260860 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:14.260918 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:14.264000 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.265026 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.265030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.265033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.265164 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.265718 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.265721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.265723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.265995 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:14.266108 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.267077 (+ 969us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.267256 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260925 10:30:14.269222 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.247028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:39551 (local address 127.12.248.66:38645)
0925 10:30:14.247766 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:14.247810 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:14.249715 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:14.250512 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:14.250520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:14.251137 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:14.251896 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.251918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.253127 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.253132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.255357 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.255378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.256431 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.256440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.256795 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.257607 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:14.257644 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:14.260514 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:14.264160 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:14.264174 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:14.264196 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:14.264890 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:14.265277 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:14.265285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:14.265291 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:14.265594 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:14.266141 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:14.266152 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:14.266659 (+ 507us) client_negotiation.cc:769] Sending connection context
0925 10:30:14.266992 (+ 333us) client_negotiation.cc:240] Negotiation successful
0925 10:30:14.267672 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":290,"threads_started":1}
I20260925 10:30:14.271024 30842 heartbeater.cc:347] Connected to a master server at 127.12.248.126:39551
I20260925 10:30:14.271389 30842 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:14.272289 30842 heartbeater.cc:511] Master 127.12.248.126:39551 requested a full tablet report, sending...
I20260925 10:30:14.274142 30516 ts_manager.cc:194] Registered new tserver with Master: b7ebe4fba19241a3b098706c2040126f (127.12.248.66:33169)
I20260925 10:30:14.275463 30516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:38645
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:30:14.313004 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:14.659004 30848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:14.659405 30848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:14.659533 30848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:14.667572 30848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:14.667681 30848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:14.667794 30848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:14.677609 30848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:39551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:14.679131 30848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:14.680591 30848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:14.693909 30853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:14.693945 30856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:14.693993 30854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:14.699177 30848 server_base.cc:1034] running on GCE node
I20260925 10:30:14.699883 30848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:14.700924 30848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:14.702189 30848 hybrid_clock.cc:648] HybridClock initialized: now 1790332214702137 us; error 43 us; skew 500 ppm
I20260925 10:30:14.702596 30848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:14 dist-test-slave-f9rl krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332214, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:14.705893 30848 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:30:14.707399 30848 webserver.cc:460] Webserver started at http://127.12.248.67:34661/ using document root <none> and password file <none>
I20260925 10:30:14.708138 30848 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:14.708285 30848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:14.708611 30848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:14.711802 30848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "55ad1b49ba484a94a8fa4676125881e7"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "9600683dbfa21f882aff963278bfb12c"
server_key_iv: "1bc362924df3130f5d2d4879a77bcf7d"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.712556 30848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "55ad1b49ba484a94a8fa4676125881e7"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "9600683dbfa21f882aff963278bfb12c"
server_key_iv: "1bc362924df3130f5d2d4879a77bcf7d"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.718062 30848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:30:14.721668 30863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.723186 30848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:14.723366 30848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "55ad1b49ba484a94a8fa4676125881e7"
format_stamp: "Formatted at 2026-09-25 10:30:14 on dist-test-slave-f9rl"
server_key: "9600683dbfa21f882aff963278bfb12c"
server_key_iv: "1bc362924df3130f5d2d4879a77bcf7d"
server_key_version: "encryptionkey@0"
I20260925 10:30:14.723569 30848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:14.745828 30707 heartbeater.cc:503] Master 127.12.248.126:39551 was elected leader, sending a full tablet report...
I20260925 10:30:14.746642 30848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:14.751163 30848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:14.751459 30848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:14.752458 30848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:14.753908 30848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:14.754014 30848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.754163 30848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:14.754240 30848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:14.788770 30848 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:33069
I20260925 10:30:14.788817 30976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:33069 every 8 connection(s)
I20260925 10:30:14.790211 30848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:14.798666 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 30848
I20260925 10:30:14.798862 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:14.799420 13281 external_mini_cluster.cc:1428] Setting key bc2a4217958835a200d5bc1852959b06
Sep 25 10:30:14 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332214, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:14.811800 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.792891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:60951 (local address 127.12.248.126:39551)
0925 10:30:14.793407 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.793414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.794400 (+ 986us) server_negotiation.cc:407] Connection header received
0925 10:30:14.796343 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.796347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.796415 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.796513 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:14.798155 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.799392 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.801358 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.802003 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.802167 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.805879 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:14.805885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:14.805888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:14.805919 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:14.808412 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.809293 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.809298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.809303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.809422 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.809938 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.809943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.809947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.810303 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:14.810469 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.811182 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.811404 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":233,"threads_started":1}
I20260925 10:30:14.812785 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.793478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:39551 (local address 127.12.248.67:60951)
0925 10:30:14.794234 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:14.794276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:14.796087 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:14.796767 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:14.796776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:14.797336 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:14.797975 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.797996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.799577 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.799584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.801200 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.801213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.802259 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.802267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.802570 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.803387 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:14.803410 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:14.805657 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:14.808668 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:14.808683 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:14.808697 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:14.809168 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:14.809596 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:14.809605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:14.809608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:14.809795 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:14.810435 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:14.810444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:14.810798 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 10:30:14.811065 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 10:30:14.811602 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":268,"threads_started":1}
I20260925 10:30:14.814718 30977 heartbeater.cc:347] Connected to a master server at 127.12.248.126:39551
I20260925 10:30:14.815100 30977 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:14.816032 30977 heartbeater.cc:511] Master 127.12.248.126:39551 requested a full tablet report, sending...
I20260925 10:30:14.817899 30515 ts_manager.cc:194] Registered new tserver with Master: 55ad1b49ba484a94a8fa4676125881e7 (127.12.248.67:33069)
I20260925 10:30:14.818858 30515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:60951
I20260925 10:30:14.828694 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:14.848510 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.833890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.12.248.126:39551)
0925 10:30:14.834140 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.834150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.834543 (+ 393us) server_negotiation.cc:407] Connection header received
0925 10:30:14.834733 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.834737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.834806 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.834894 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:14.836525 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.837307 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.838989 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.839411 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.839512 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.842379 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:14.842384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:14.842387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:14.842423 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:14.845222 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.846115 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.846119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.846122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.846280 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:14.846774 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:14.846778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:14.846780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:14.847166 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:14.847366 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.847893 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.848183 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:30:14.853719 30515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:14.861198 30515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:14.889758 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.876441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.12.248.66:33169)
0925 10:30:14.879095 (+ 2654us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.879106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.879148 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:30:14.879296 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.879306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.879639 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.880695 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:14.881017 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.883467 (+ 2450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.886357 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.888042 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.888194 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.888441 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.889158 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.889354 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2389,"thread_start_us":2263,"threads_started":1}
I20260925 10:30:14.889786 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.875861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:33169 (local address 127.0.0.1:34120)
0925 10:30:14.877540 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:14.877597 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:14.877801 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:14.879932 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:14.879940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:14.879975 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:14.880416 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.880433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.883684 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.883693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.886137 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.886152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.888618 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.888625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.888857 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.888915 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:30:14.889064 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:30:14.889280 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"mutex_wait_us":36,"thread_start_us":1258,"threads_started":1}
I20260925 10:30:14.891661 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.876058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.12.248.65:44069)
0925 10:30:14.879173 (+ 3115us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.879184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.879872 (+ 688us) server_negotiation.cc:407] Connection header received
0925 10:30:14.879993 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.880002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.880243 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.880459 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:14.881184 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.883099 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.888648 (+ 5549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.890374 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.890555 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.890823 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.891145 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.891323 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2918,"thread_start_us":2173,"threads_started":1}
I20260925 10:30:14.891710 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.875973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:44069 (local address 127.0.0.1:41100)
0925 10:30:14.879712 (+ 3739us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:14.879733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:14.879862 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:14.880441 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:14.880445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:14.880474 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:14.880913 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.880926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.883291 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.883297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.888432 (+ 5135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.888445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.890618 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.890625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.890917 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.890942 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:30:14.891088 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:30:14.891245 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3535,"mutex_wait_us":206,"thread_start_us":3359,"threads_started":1}
I20260925 10:30:14.894486 30776 tablet_service.cc:1467] Processing CreateTablet for tablet 71103efd23c34208bf901a510112d283 (DEFAULT_TABLE table=test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:14.895181 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.880859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:33069 (local address 127.0.0.1:59568)
0925 10:30:14.885303 (+ 4444us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:14.885332 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:14.885543 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:14.887111 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:14.887117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:14.887150 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:14.887700 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.887725 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.890167 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.890172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.892410 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:14.892426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.894310 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.894316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:14.894495 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.894529 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:30:14.894643 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:30:14.894779 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4201,"mutex_wait_us":1,"thread_start_us":3915,"threads_started":1}
I20260925 10:30:14.895223 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:14.880230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.12.248.67:33069)
0925 10:30:14.886296 (+ 6066us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:14.886306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:14.886343 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:30:14.886461 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:14.886471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:14.886831 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:14.887925 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:14.888296 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.889982 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.892614 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:14.894088 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:14.894221 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:14.894443 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:14.894711 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 10:30:14.894880 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5780,"thread_start_us":5605,"threads_started":1}
I20260925 10:30:14.895519 30641 tablet_service.cc:1467] Processing CreateTablet for tablet 71103efd23c34208bf901a510112d283 (DEFAULT_TABLE table=test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:14.896158 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71103efd23c34208bf901a510112d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:14.896965 30641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71103efd23c34208bf901a510112d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:14.898940 30911 tablet_service.cc:1467] Processing CreateTablet for tablet 71103efd23c34208bf901a510112d283 (DEFAULT_TABLE table=test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:14.900345 30911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71103efd23c34208bf901a510112d283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:14.908717 30996 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Bootstrap starting.
I20260925 10:30:14.912163 30997 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Bootstrap starting.
I20260925 10:30:14.912369 30996 tablet_bootstrap.cc:654] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:14.913829 30996 log.cc:824] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:14.915813 30998 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Bootstrap starting.
I20260925 10:30:14.915902 30997 tablet_bootstrap.cc:654] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:14.917217 30997 log.cc:824] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:14.918424 30996 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: No bootstrap required, opened a new log
I20260925 10:30:14.918727 30996 ts_tablet_manager.cc:1397] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260925 10:30:14.919984 30998 tablet_bootstrap.cc:654] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:14.920522 30997 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: No bootstrap required, opened a new log
I20260925 10:30:14.920806 30997 ts_tablet_manager.cc:1397] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:30:14.921262 30998 log.cc:824] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:14.924371 30998 tablet_bootstrap.cc:492] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: No bootstrap required, opened a new log
I20260925 10:30:14.924669 30998 ts_tablet_manager.cc:1397] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 10:30:14.927220 30996 raft_consensus.cc:348] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.927716 30996 raft_consensus.cc:374] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:14.927831 30996 raft_consensus.cc:729] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ebe4fba19241a3b098706c2040126f, State: Initialized, Role: FOLLOWER
I20260925 10:30:14.928514 30996 consensus_queue.cc:260] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.929744 30842 heartbeater.cc:503] Master 127.12.248.126:39551 was elected leader, sending a full tablet report...
I20260925 10:30:14.930442 30996 ts_tablet_manager.cc:1428] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260925 10:30:14.931835 30997 raft_consensus.cc:348] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.932299 30997 raft_consensus.cc:374] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:14.932403 30997 raft_consensus.cc:729] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56015e9a11394c9f9b3f7ec37802a616, State: Initialized, Role: FOLLOWER
I20260925 10:30:14.933039 30997 consensus_queue.cc:260] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.934857 30997 ts_tablet_manager.cc:1428] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260925 10:30:14.935652 30998 raft_consensus.cc:348] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.936175 30998 raft_consensus.cc:374] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:14.936298 30998 raft_consensus.cc:729] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ad1b49ba484a94a8fa4676125881e7, State: Initialized, Role: FOLLOWER
I20260925 10:30:14.936931 30998 consensus_queue.cc:260] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:14.938217 30977 heartbeater.cc:503] Master 127.12.248.126:39551 was elected leader, sending a full tablet report...
I20260925 10:30:14.938831 30998 ts_tablet_manager.cc:1428] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260925 10:30:14.963186 30708 tablet.cc:2369] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:30:14.995857 30843 tablet.cc:2369] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:30:15.041903 30978 tablet.cc:2369] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:15.248113 31004 raft_consensus.cc:482] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:15.248471 31004 raft_consensus.cc:504] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:15.250581 31004 leader_election.cc:288] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56015e9a11394c9f9b3f7ec37802a616 (127.12.248.65:44069), b7ebe4fba19241a3b098706c2040126f (127.12.248.66:33169)
I20260925 10:30:15.257681 30980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.250870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:33169 (local address 127.12.248.67:41009)
0925 10:30:15.251039 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:15.251059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:15.251191 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:15.252259 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:15.252265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:15.252305 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:15.252540 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:15.252555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.253978 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.253984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:15.255729 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:15.255741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.256899 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.256905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:15.257088 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.257120 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:30:15.257230 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:30:15.257366 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260925 10:30:15.257671 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.251582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:41009 (local address 127.12.248.66:33169)
0925 10:30:15.251844 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.251851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.251874 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:30:15.251962 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.251970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.252071 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.252208 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:15.252737 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.253840 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.255910 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.256735 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.256906 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.256978 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.257258 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.257385 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 10:30:15.259055 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.250870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:44069 (local address 127.12.248.67:40609)
0925 10:30:15.252787 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:15.252802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:15.252950 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:15.253291 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:15.253295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:15.253320 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:15.253708 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:15.253718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.255149 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.255154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:15.256739 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:15.256753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.258349 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.258355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:15.258482 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.258504 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:30:15.258578 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:30:15.258735 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"mutex_wait_us":869,"thread_start_us":753,"threads_started":1}
I20260925 10:30:15.259092 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.251057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:40609 (local address 127.12.248.65:44069)
0925 10:30:15.251309 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.251315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.252896 (+ 1581us) server_negotiation.cc:407] Connection header received
0925 10:30:15.253100 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.253105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.253172 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.253272 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:15.253868 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.254967 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.256937 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.258123 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.258308 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.258402 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.258643 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.258793 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:30:15.258852 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71103efd23c34208bf901a510112d283" candidate_uuid: "55ad1b49ba484a94a8fa4676125881e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebe4fba19241a3b098706c2040126f" is_pre_election: true
I20260925 10:30:15.259516 30796 raft_consensus.cc:2454] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ad1b49ba484a94a8fa4676125881e7 in term 0.
I20260925 10:30:15.260341 30661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71103efd23c34208bf901a510112d283" candidate_uuid: "55ad1b49ba484a94a8fa4676125881e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56015e9a11394c9f9b3f7ec37802a616" is_pre_election: true
I20260925 10:30:15.260658 30866 leader_election.cc:302] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ad1b49ba484a94a8fa4676125881e7, b7ebe4fba19241a3b098706c2040126f; no voters:
I20260925 10:30:15.260973 30661 raft_consensus.cc:2454] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ad1b49ba484a94a8fa4676125881e7 in term 0.
I20260925 10:30:15.261168 31004 raft_consensus.cc:2781] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:15.261354 31004 raft_consensus.cc:482] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:15.261538 31004 raft_consensus.cc:3037] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:15.264020 31004 raft_consensus.cc:504] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:15.264771 31004 leader_election.cc:288] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [CANDIDATE]: Term 1 election: Requested vote from peers 56015e9a11394c9f9b3f7ec37802a616 (127.12.248.65:44069), b7ebe4fba19241a3b098706c2040126f (127.12.248.66:33169)
I20260925 10:30:15.265316 30661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71103efd23c34208bf901a510112d283" candidate_uuid: "55ad1b49ba484a94a8fa4676125881e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56015e9a11394c9f9b3f7ec37802a616"
I20260925 10:30:15.265345 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71103efd23c34208bf901a510112d283" candidate_uuid: "55ad1b49ba484a94a8fa4676125881e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ebe4fba19241a3b098706c2040126f"
I20260925 10:30:15.265663 30796 raft_consensus.cc:3037] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:15.265681 30661 raft_consensus.cc:3037] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:15.268141 30796 raft_consensus.cc:2454] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ad1b49ba484a94a8fa4676125881e7 in term 1.
I20260925 10:30:15.268164 30661 raft_consensus.cc:2454] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ad1b49ba484a94a8fa4676125881e7 in term 1.
I20260925 10:30:15.268648 30866 leader_election.cc:302] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ad1b49ba484a94a8fa4676125881e7, b7ebe4fba19241a3b098706c2040126f; no voters:
I20260925 10:30:15.269025 31004 raft_consensus.cc:2781] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:15.269661 31004 raft_consensus.cc:686] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 1 LEADER]: Becoming Leader. State: Replica: 55ad1b49ba484a94a8fa4676125881e7, State: Running, Role: LEADER
I20260925 10:30:15.270191 31004 consensus_queue.cc:237] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } }
I20260925 10:30:15.275484 30515 catalog_manager.cc:5495] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ad1b49ba484a94a8fa4676125881e7 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "55ad1b49ba484a94a8fa4676125881e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55ad1b49ba484a94a8fa4676125881e7" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 33069 } health_report { overall_health: HEALTHY } } }
I20260925 10:30:15.344843 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.339011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.12.248.67:33069)
0925 10:30:15.339262 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.339268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.339291 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:30:15.339409 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.339413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.339500 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.339663 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:15.340093 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.340836 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.342494 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.342893 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.343037 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.343583 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:15.344260 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:15.344423 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.344480 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.344564 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:30:15.354516 30661 raft_consensus.cc:1261] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 1 FOLLOWER]: Refusing update from remote peer 55ad1b49ba484a94a8fa4676125881e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:15.354508 30796 raft_consensus.cc:1261] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 1 FOLLOWER]: Refusing update from remote peer 55ad1b49ba484a94a8fa4676125881e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:15.355893 31006 consensus_queue.cc:1035] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ebe4fba19241a3b098706c2040126f" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:15.356352 31004 consensus_queue.cc:1035] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56015e9a11394c9f9b3f7ec37802a616" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:15.377000 31010 mvcc.cc:205] Tried to move back new op lower bound from 7333200754079723520 to 7333200753751162880. Current Snapshot: MvccSnapshot[applied={T|T < 7333200754079723520}]
I20260925 10:30:15.385955 30515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:15.386255 30515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:15.389825 30515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c7f9b473dcb4c66a2bb98c45eae40c5: Sending DeleteTablet for 3 replicas of tablet 71103efd23c34208bf901a510112d283
I20260925 10:30:15.391273 30641 tablet_service.cc:1514] Processing DeleteTablet for tablet 71103efd23c34208bf901a510112d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:15 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:41100
I20260925 10:30:15.391247 30911 tablet_service.cc:1514] Processing DeleteTablet for tablet 71103efd23c34208bf901a510112d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:15 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:59568
I20260925 10:30:15.391247 30776 tablet_service.cc:1514] Processing DeleteTablet for tablet 71103efd23c34208bf901a510112d283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:15 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:34120
I20260925 10:30:15.394984 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30578
I20260925 10:30:15.404989 31021 tablet_replica.cc:330] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: stopping tablet replica
I20260925 10:30:15.405684 31021 raft_consensus.cc:2229] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:15.406584 31021 raft_consensus.cc:2258] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:15.407904 31020 tablet_replica.cc:330] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: stopping tablet replica
I20260925 10:30:15.408524 31020 raft_consensus.cc:2229] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:15.409240 31020 raft_consensus.cc:2258] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:15.411801 31020 ts_tablet_manager.cc:1905] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:15.411810 31021 ts_tablet_manager.cc:1905] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:15.420104 31020 ts_tablet_manager.cc:1918] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:15.420320 31020 log.cc:1196] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/71103efd23c34208bf901a510112d283
I20260925 10:30:15.420997 31020 ts_tablet_manager.cc:1939] T 71103efd23c34208bf901a510112d283 P b7ebe4fba19241a3b098706c2040126f: Deleting consensus metadata
I20260925 10:30:15.423246 30502 catalog_manager.cc:4841] TS b7ebe4fba19241a3b098706c2040126f (127.12.248.66:33169): tablet 71103efd23c34208bf901a510112d283 (table test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]) successfully deleted
I20260925 10:30:15.427347 31022 tablet_replica.cc:330] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: stopping tablet replica
I20260925 10:30:15.427912 31022 raft_consensus.cc:2229] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:15.428574 31022 raft_consensus.cc:2258] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:15.428866 31021 ts_tablet_manager.cc:1918] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:15.429065 31021 log.cc:1196] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/71103efd23c34208bf901a510112d283
I20260925 10:30:15.430816 31022 ts_tablet_manager.cc:1905] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:15.433014 31021 ts_tablet_manager.cc:1939] T 71103efd23c34208bf901a510112d283 P 55ad1b49ba484a94a8fa4676125881e7: Deleting consensus metadata
Sep 25 10:30:15 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:30:15.435185 30500 catalog_manager.cc:4841] TS 55ad1b49ba484a94a8fa4676125881e7 (127.12.248.67:33069): tablet 71103efd23c34208bf901a510112d283 (table test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]) successfully deleted
I20260925 10:30:15.438367 31022 ts_tablet_manager.cc:1918] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:15.438609 31022 log.cc:1196] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/71103efd23c34208bf901a510112d283
I20260925 10:30:15.439216 31022 ts_tablet_manager.cc:1939] T 71103efd23c34208bf901a510112d283 P 56015e9a11394c9f9b3f7ec37802a616: Deleting consensus metadata
I20260925 10:30:15.441318 30500 catalog_manager.cc:4841] TS 56015e9a11394c9f9b3f7ec37802a616 (127.12.248.65:44069): tablet 71103efd23c34208bf901a510112d283 (table test-table [id=43bc2e5505eb4333a3e7b44ed58ca6dd]) successfully deleted
I20260925 10:30:15.442947 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.397167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.12.248.65:44069)
0925 10:30:15.397424 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.397453 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.415894 (+ 18441us) server_negotiation.cc:407] Connection header received
0925 10:30:15.416513 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.416520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.416634 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.416725 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:15.423013 (+ 6288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.424277 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.426510 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.427362 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.427494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.434742 (+ 7248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:15.434759 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:15.434779 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:15.434835 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:15.439395 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.440398 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.440408 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.440412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.440607 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.441121 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.441126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.441129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.441655 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:15.441964 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.442280 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.442550 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:30:15.443773 30702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:15.560559 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30578
I20260925 10:30:15.579283 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30713
Sep 25 10:30:15 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:30:15.598484 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.580283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.12.248.66:33169)
0925 10:30:15.580503 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.580510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.580534 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:30:15.580614 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.580619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.580704 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.580820 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:15.582570 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.583689 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.585283 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.586064 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.586194 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.591054 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:15.591074 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:15.591096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:15.591151 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:15.595169 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.596029 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.596039 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.596043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.596209 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.596615 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.596620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.596624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.597133 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:15.597465 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.597851 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.598094 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:30:15.599363 30837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:15.696696 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30713
I20260925 10:30:15.717052 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30848
Sep 25 10:30:15 dist-test-slave-f9rl krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:30:15.735773 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:15.718085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.12.248.67:33069)
0925 10:30:15.718330 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:15.718341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:15.718363 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:30:15.718432 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:15.718437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:15.718537 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:15.718654 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:15.719976 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.721152 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.722621 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:15.723294 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:15.723445 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:15.728272 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:15.728295 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:15.728318 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:15.728368 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:15.732427 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.733111 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.733121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.733125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.733341 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:15.733902 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:15.733908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:15.733911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:15.734423 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:15.734761 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:15.735128 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:30:15.735373 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:30:15.736564 30972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:15.863045 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30848
I20260925 10:30:15.883886 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 30484
I20260925 10:30:15.885339 30547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:16.003363 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 30484
2026-09-25T10:30:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4780 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:16Z Disabled control of system clock
I20260925 10:30:16.084744 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36175
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:16.437742 31031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:16.438069 31031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:16.438159 31031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:16.445948 31031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:16.446086 31031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:16.446131 31031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:16.446193 31031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:16.446260 31031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:16.456118 31031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:36175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:16.457492 31031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:16.459367 31031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:16.472333 31036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:16.472308 31037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:16.472415 31039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:16.473244 31031 server_base.cc:1034] running on GCE node
I20260925 10:30:16.474126 31031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:16.475605 31031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:16.476819 31031 hybrid_clock.cc:648] HybridClock initialized: now 1790332216476779 us; error 28 us; skew 500 ppm
I20260925 10:30:16.477216 31031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:16.484694 31031 webserver.cc:460] Webserver started at http://127.12.248.126:38647/ using document root <none> and password file <none>
I20260925 10:30:16.485558 31031 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:16.485683 31031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:16.486069 31031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:16.489154 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "4b7c0196a0374c43b651df75852f6749"
format_stamp: "Formatted at 2026-09-25 10:30:16 on dist-test-slave-f9rl"
server_key: "4606df394710ad75bca9c356195707ba"
server_key_iv: "746a0ca93194ecf9be660a8ef341b577"
server_key_version: "encryptionkey@0"
I20260925 10:30:16.490124 31031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "4b7c0196a0374c43b651df75852f6749"
format_stamp: "Formatted at 2026-09-25 10:30:16 on dist-test-slave-f9rl"
server_key: "4606df394710ad75bca9c356195707ba"
server_key_iv: "746a0ca93194ecf9be660a8ef341b577"
server_key_version: "encryptionkey@0"
I20260925 10:30:16.495569 31031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:16.499331 31045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:16.500972 31031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:30:16.501178 31031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "4b7c0196a0374c43b651df75852f6749"
format_stamp: "Formatted at 2026-09-25 10:30:16 on dist-test-slave-f9rl"
server_key: "4606df394710ad75bca9c356195707ba"
server_key_iv: "746a0ca93194ecf9be660a8ef341b577"
server_key_version: "encryptionkey@0"
I20260925 10:30:16.501403 31031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:16.516355 31031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:16.517299 31031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:16.517659 31031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:16.536626 31031 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:36175
I20260925 10:30:16.536686 31096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:36175 every 8 connection(s)
I20260925 10:30:16.538106 31031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:16.541478 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31031
I20260925 10:30:16.541723 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:16.542238 13281 external_mini_cluster.cc:1428] Setting key 6c2cf5136d3a875f9683e97c337d2d90
I20260925 10:30:16.542312 31097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:16.558514 31097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: Bootstrap starting.
I20260925 10:30:16.558594 31100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:16.544431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.12.248.126:36175)
0925 10:30:16.545683 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:16.545697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:16.545755 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:30:16.547611 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:16.547639 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:16.548286 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:16.548772 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:16.549352 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:16.551461 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:16.553041 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:16.554159 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:16.554364 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:16.555629 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:16.555640 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:16.555659 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:16.556406 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:16.556515 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:16.556541 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:16.556652 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:16.556919 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 10:30:16.557253 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":425,"threads_started":1}
I20260925 10:30:16.561977 31097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:16.563067 31097 log.cc:824] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:16.566399 31097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: No bootstrap required, opened a new log
I20260925 10:30:16.574456 31097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } }
I20260925 10:30:16.574893 31097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:16.575006 31097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7c0196a0374c43b651df75852f6749, State: Initialized, Role: FOLLOWER
I20260925 10:30:16.575639 31097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } }
I20260925 10:30:16.575922 31097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:16.576056 31097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:16.576236 31097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:16.579053 31097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } }
I20260925 10:30:16.579605 31097 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7c0196a0374c43b651df75852f6749; no voters:
I20260925 10:30:16.580283 31097 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:16.580513 31102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:16.581380 31102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7c0196a0374c43b651df75852f6749, State: Running, Role: LEADER
I20260925 10:30:16.581982 31102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } }
I20260925 10:30:16.582569 31097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:16.584659 31104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7c0196a0374c43b651df75852f6749. Latest consensus state: current_term: 1 leader_uuid: "4b7c0196a0374c43b651df75852f6749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } } }
I20260925 10:30:16.585147 31104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:16.584978 31103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7c0196a0374c43b651df75852f6749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7c0196a0374c43b651df75852f6749" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36175 } } }
I20260925 10:30:16.585248 31103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:16.586223 31108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:16.593173 31108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:16.608404 31108 catalog_manager.cc:1347] Generated new cluster ID: d78276c1ae43418484cc97bbd94c50f1
I20260925 10:30:16.608557 31108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:16.650480 31108 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:16.652443 31108 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:16.666715 31108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: Generated new TSK 0
I20260925 10:30:16.667809 31108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:16.673148 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--builtin_ntp_servers=127.12.248.84:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:17.026409 31121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:17.026708 31121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:17.026871 31121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:17.034214 31121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:17.034330 31121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:17.034513 31121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:17.043884 31121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:17.045246 31121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:17.047250 31121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:17.061631 31126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:17.061842 31129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:17.062040 31121 server_base.cc:1034] running on GCE node
W20260925 10:30:17.063367 31127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:17.064072 31121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:17.065295 31121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:17.066608 31121 hybrid_clock.cc:648] HybridClock initialized: now 1790332217066475 us; error 67 us; skew 500 ppm
I20260925 10:30:17.066977 31121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:17.069623 31121 webserver.cc:460] Webserver started at http://127.12.248.65:34131/ using document root <none> and password file <none>
I20260925 10:30:17.070324 31121 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:17.070461 31121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:17.070785 31121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:17.073834 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "d080209807c64ed7a46e98f193321e0c"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "eb83607124e403da5901bf7c2dd94e2f"
server_key_iv: "3363aa74c9b73789647d528e8db46cc6"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.074676 31121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "d080209807c64ed7a46e98f193321e0c"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "eb83607124e403da5901bf7c2dd94e2f"
server_key_iv: "3363aa74c9b73789647d528e8db46cc6"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.079949 31121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:17.084209 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.085973 31121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:17.086189 31121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "d080209807c64ed7a46e98f193321e0c"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "eb83607124e403da5901bf7c2dd94e2f"
server_key_iv: "3363aa74c9b73789647d528e8db46cc6"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.086460 31121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:17.104101 31121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:17.105295 31121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:17.105702 31121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:17.106909 31121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:17.108539 31121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:17.108675 31121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.108819 31121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:17.108909 31121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.144599 31121 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:36235
I20260925 10:30:17.144665 31248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:36235 every 8 connection(s)
I20260925 10:30:17.146085 31121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:17.150655 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31121
I20260925 10:30:17.150844 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:17.151360 13281 external_mini_cluster.cc:1428] Setting key c1a94a5b0ece29f0732b955607f36405
I20260925 10:30:17.154323 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--builtin_ntp_servers=127.12.248.84:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:17.160684 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:17.149063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:49201 (local address 127.12.248.126:36175)
0925 10:30:17.149579 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:17.149585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:17.151070 (+ 1485us) server_negotiation.cc:407] Connection header received
0925 10:30:17.153129 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:17.153135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:17.153221 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:17.153357 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:17.154547 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.155782 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.157397 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.158156 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.158306 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:17.159701 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:17.159710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:17.159717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:17.159745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:17.159803 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:17.159821 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:17.159916 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:17.160123 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:30:17.160240 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":206,"threads_started":1}
I20260925 10:30:17.161675 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:17.149811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:36175 (local address 127.12.248.65:49201)
0925 10:30:17.150865 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:17.150930 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:17.152847 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:17.153629 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:17.153638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:17.153739 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:17.154332 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:17.154349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.155940 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.155943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:17.157248 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:17.157260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.158378 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.158382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:17.158578 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:17.159358 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:17.159361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:17.159407 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:17.159485 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:17.159920 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:17.159928 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:17.159934 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:30:17.160101 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 10:30:17.160406 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":351,"threads_started":1}
I20260925 10:30:17.163317 31249 heartbeater.cc:347] Connected to a master server at 127.12.248.126:36175
I20260925 10:30:17.163713 31249 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:17.164492 31249 heartbeater.cc:511] Master 127.12.248.126:36175 requested a full tablet report, sending...
I20260925 10:30:17.166752 31062 ts_manager.cc:194] Registered new tserver with Master: d080209807c64ed7a46e98f193321e0c (127.12.248.65:36235)
I20260925 10:30:17.168246 31062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.65:49201
W20260925 10:30:17.501217 31253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:17.501591 31253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:17.501782 31253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:17.509346 31253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:17.509527 31253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:17.509722 31253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:17.519126 31253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:17.520568 31253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:17.522737 31253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:17.536460 31258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:17.536444 31259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:17.537053 31253 server_base.cc:1034] running on GCE node
W20260925 10:30:17.537688 31261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:17.538492 31253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:17.539654 31253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:17.540915 31253 hybrid_clock.cc:648] HybridClock initialized: now 1790332217540868 us; error 52 us; skew 500 ppm
I20260925 10:30:17.541293 31253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:17.544183 31253 webserver.cc:460] Webserver started at http://127.12.248.66:34233/ using document root <none> and password file <none>
I20260925 10:30:17.545058 31253 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:17.545172 31253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:17.545553 31253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:17.549515 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "79cebb83666686e90f68c9b9af65892d"
server_key_iv: "43b28e60248b7b922b2f0be9d30754e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.550607 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "79cebb83666686e90f68c9b9af65892d"
server_key_iv: "43b28e60248b7b922b2f0be9d30754e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.557560 31253 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 10:30:17.561703 31267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.563549 31253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:17.563699 31253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac"
format_stamp: "Formatted at 2026-09-25 10:30:17 on dist-test-slave-f9rl"
server_key: "79cebb83666686e90f68c9b9af65892d"
server_key_iv: "43b28e60248b7b922b2f0be9d30754e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:17.563880 31253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:17.580984 31253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:17.581892 31253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:17.582161 31253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:17.583209 31253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:17.584772 31253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:17.584872 31253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.584946 31253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:17.585000 31253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:17.619688 31253 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:33563
I20260925 10:30:17.619771 31380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:33563 every 8 connection(s)
I20260925 10:30:17.621130 31253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:17.622051 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31253
I20260925 10:30:17.622268 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:17.622833 13281 external_mini_cluster.cc:1428] Setting key 53e491a94c4cacc32542e393854fa307
I20260925 10:30:17.626379 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--builtin_ntp_servers=127.12.248.84:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:17.637274 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:17.624138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:39531 (local address 127.12.248.126:36175)
0925 10:30:17.624313 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:17.624317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:17.626241 (+ 1924us) server_negotiation.cc:407] Connection header received
0925 10:30:17.629126 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:17.629132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:17.629199 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:17.629288 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:17.630761 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.631459 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.634055 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.634456 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.634640 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:17.636267 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:17.636276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:17.636284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:17.636312 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:17.636358 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:17.636377 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:17.636472 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:17.636732 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 10:30:17.636848 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:30:17.638154 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:17.624849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:36175 (local address 127.12.248.66:39531)
0925 10:30:17.626054 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:17.626111 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:17.628815 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:17.629583 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:17.629597 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:17.629728 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:17.630572 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:17.630597 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.631662 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.631671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:17.633874 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:17.633892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:17.634775 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:17.634785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:17.635126 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:17.635889 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:17.635891 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:17.635933 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:17.636012 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:17.636479 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:17.636487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:17.636493 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:30:17.636662 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 10:30:17.636975 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":333,"threads_started":1}
I20260925 10:30:17.639654 31381 heartbeater.cc:347] Connected to a master server at 127.12.248.126:36175
I20260925 10:30:17.639988 31381 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:17.640938 31381 heartbeater.cc:511] Master 127.12.248.126:36175 requested a full tablet report, sending...
I20260925 10:30:17.642580 31062 ts_manager.cc:194] Registered new tserver with Master: 7bb0e3351f284cbbbfa4dd4fd17525ac (127.12.248.66:33563)
I20260925 10:30:17.643410 31062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.66:39531
W20260925 10:30:17.978760 31384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:17.979069 31384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:17.979235 31384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:17.988682 31384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:17.988798 31384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:17.988986 31384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:17.998883 31384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:18.000169 31384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:18.002005 31384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:18.015697 31389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:18.015745 31390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:18.016224 31392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:18.016496 31384 server_base.cc:1034] running on GCE node
I20260925 10:30:18.017073 31384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:18.018146 31384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:18.019349 31384 hybrid_clock.cc:648] HybridClock initialized: now 1790332218019284 us; error 55 us; skew 500 ppm
I20260925 10:30:18.019647 31384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:18.022086 31384 webserver.cc:460] Webserver started at http://127.12.248.67:41917/ using document root <none> and password file <none>
I20260925 10:30:18.022759 31384 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:18.022892 31384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:18.023227 31384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:18.026187 31384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "6d44548902174353ae21ef247fb787e2"
format_stamp: "Formatted at 2026-09-25 10:30:18 on dist-test-slave-f9rl"
server_key: "b52b69c69af7822e4ed8962efa632a06"
server_key_iv: "a1434d20c001df17e9fb9a6c9032ca2c"
server_key_version: "encryptionkey@0"
I20260925 10:30:18.026993 31384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "6d44548902174353ae21ef247fb787e2"
format_stamp: "Formatted at 2026-09-25 10:30:18 on dist-test-slave-f9rl"
server_key: "b52b69c69af7822e4ed8962efa632a06"
server_key_iv: "a1434d20c001df17e9fb9a6c9032ca2c"
server_key_version: "encryptionkey@0"
I20260925 10:30:18.032272 31384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:30:18.036070 31398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:18.037647 31384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 10:30:18.037823 31384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "6d44548902174353ae21ef247fb787e2"
format_stamp: "Formatted at 2026-09-25 10:30:18 on dist-test-slave-f9rl"
server_key: "b52b69c69af7822e4ed8962efa632a06"
server_key_iv: "a1434d20c001df17e9fb9a6c9032ca2c"
server_key_version: "encryptionkey@0"
I20260925 10:30:18.038046 31384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:18.058507 31384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:18.059341 31384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:18.059655 31384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:18.060662 31384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:18.062283 31384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:18.062407 31384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:18.062537 31384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:18.062642 31384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:18.097915 31384 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:45481
I20260925 10:30:18.097970 31511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:45481 every 8 connection(s)
I20260925 10:30:18.099433 31384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:18.107412 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31384
I20260925 10:30:18.107617 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:18.108206 13281 external_mini_cluster.cc:1428] Setting key 9f0143ecb0dda80464f2bc04d049002c
I20260925 10:30:18.113679 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.102255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:49299 (local address 127.12.248.126:36175)
0925 10:30:18.102483 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.102490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.103938 (+ 1448us) server_negotiation.cc:407] Connection header received
0925 10:30:18.105926 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.105930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.106006 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.106170 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:18.107319 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.108489 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.110601 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.111309 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.111450 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.112829 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:18.112833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:18.112837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:18.112855 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:18.112886 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:18.112898 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:18.112970 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.113215 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.113299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:30:18.114773 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.102811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:36175 (local address 127.12.248.67:49299)
0925 10:30:18.103764 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.103811 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.105626 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.106392 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.106401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.106495 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:18.107108 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.107125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.108693 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.108697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.110460 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.110472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.111588 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.111595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.111812 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.112458 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:18.112460 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:18.112502 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:18.112590 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:18.112987 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:18.112995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:18.113001 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.113158 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.113543 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
I20260925 10:30:18.116317 31512 heartbeater.cc:347] Connected to a master server at 127.12.248.126:36175
I20260925 10:30:18.116660 31512 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:18.117544 31512 heartbeater.cc:511] Master 127.12.248.126:36175 requested a full tablet report, sending...
I20260925 10:30:18.119095 31062 ts_manager.cc:194] Registered new tserver with Master: 6d44548902174353ae21ef247fb787e2 (127.12.248.67:45481)
I20260925 10:30:18.119930 31062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.67:49299
I20260925 10:30:18.124109 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:18.134788 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.128489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.12.248.126:36175)
0925 10:30:18.128730 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.128739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.129116 (+ 377us) server_negotiation.cc:407] Connection header received
0925 10:30:18.129347 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.129351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.129427 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.129599 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:18.130105 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.130808 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.132263 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.132695 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.132845 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.134034 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:18.134047 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:18.134051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:18.134070 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:18.134103 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:18.134115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:18.134219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.134372 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.134470 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:30:18.139031 31062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:18.147199 31062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:18.171288 31249 heartbeater.cc:503] Master 127.12.248.126:36175 was elected leader, sending a full tablet report...
I20260925 10:30:18.172268 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.161276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.12.248.65:36235)
0925 10:30:18.162166 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.162175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.163773 (+ 1598us) server_negotiation.cc:407] Connection header received
0925 10:30:18.163906 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.163917 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.164138 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.164357 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:18.165239 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166794 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.169586 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.170764 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.170848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.171133 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.171803 (+ 670us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.171985 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":376,"threads_started":1}
I20260925 10:30:18.173079 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.160412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.12.248.67:45481)
0925 10:30:18.161169 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.161183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.163217 (+ 2034us) server_negotiation.cc:407] Connection header received
0925 10:30:18.163508 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.163516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.163801 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.163968 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:18.165203 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166612 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.170143 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.171515 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.171628 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.171914 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.172575 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.172755 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":1047,"threads_started":1}
I20260925 10:30:18.173121 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.160093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:33563 (local address 127.0.0.1:35846)
0925 10:30:18.162898 (+ 2805us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.162931 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.163145 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.163985 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.163992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.164024 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:18.164503 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.164518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166612 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.166618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.169729 (+ 3111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.169744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.172022 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.172028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.172274 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.172403 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.172492 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.172697 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"mutex_wait_us":371,"thread_start_us":1679,"threads_started":1}
I20260925 10:30:18.173419 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.160479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36235 (local address 127.0.0.1:37024)
0925 10:30:18.163628 (+ 3149us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.163645 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.163765 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.164347 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.164352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.164390 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:18.165006 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.165021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166964 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.166967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.169308 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.169324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.171307 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.171316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.171514 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.171568 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.172813 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.172948 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3045,"mutex_wait_us":145,"thread_start_us":2902,"threads_started":1}
I20260925 10:30:18.173888 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.160011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:45481 (local address 127.0.0.1:50318)
0925 10:30:18.163032 (+ 3021us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.163075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.163263 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.164282 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.164289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.164318 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:18.164988 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.165002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166812 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.166819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.169728 (+ 2909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.169744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.172074 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.172079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.172372 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.172408 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.173407 (+ 999us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.173576 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2592,"mutex_wait_us":285,"thread_start_us":786,"threads_started":1}
I20260925 10:30:18.174238 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.160772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.12.248.66:33563)
0925 10:30:18.162586 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.162594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.163047 (+ 453us) server_negotiation.cc:407] Connection header received
0925 10:30:18.163460 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.163468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.163774 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.163969 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:18.164731 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.166455 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.169983 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.171515 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.171643 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.171882 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.173670 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.173850 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1650,"thread_start_us":1557,"threads_started":1}
I20260925 10:30:18.177341 31316 tablet_service.cc:1467] Processing CreateTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 (DEFAULT_TABLE table=test-table [id=2800385fd71f436b9a2c28c64136f005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:18.177547 31447 tablet_service.cc:1467] Processing CreateTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 (DEFAULT_TABLE table=test-table [id=2800385fd71f436b9a2c28c64136f005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:18.177553 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 (DEFAULT_TABLE table=test-table [id=2800385fd71f436b9a2c28c64136f005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:18.178696 31316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bee8b5d66b84dc8a837bfd9ca391657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:18.178908 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bee8b5d66b84dc8a837bfd9ca391657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:18.178908 31447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bee8b5d66b84dc8a837bfd9ca391657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:18.190694 31530 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Bootstrap starting.
I20260925 10:30:18.194550 31530 tablet_bootstrap.cc:654] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:18.195551 31530 log.cc:824] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:18.198494 31530 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: No bootstrap required, opened a new log
I20260925 10:30:18.198884 31530 ts_tablet_manager.cc:1397] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260925 10:30:18.200018 31532 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Bootstrap starting.
I20260925 10:30:18.202765 31532 tablet_bootstrap.cc:654] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:18.203866 31531 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Bootstrap starting.
I20260925 10:30:18.204070 31532 log.cc:824] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:18.207350 31532 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: No bootstrap required, opened a new log
I20260925 10:30:18.207170 31530 raft_consensus.cc:348] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.207480 31531 tablet_bootstrap.cc:654] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:18.207517 31530 raft_consensus.cc:374] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:18.207647 31532 ts_tablet_manager.cc:1397] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 10:30:18.207621 31530 raft_consensus.cc:729] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb0e3351f284cbbbfa4dd4fd17525ac, State: Initialized, Role: FOLLOWER
I20260925 10:30:18.208298 31530 consensus_queue.cc:260] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.208719 31531 log.cc:824] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:18.214486 31381 heartbeater.cc:503] Master 127.12.248.126:36175 was elected leader, sending a full tablet report...
I20260925 10:30:18.215078 31530 ts_tablet_manager.cc:1428] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260925 10:30:18.216981 31531 tablet_bootstrap.cc:492] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: No bootstrap required, opened a new log
I20260925 10:30:18.217291 31531 ts_tablet_manager.cc:1397] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.006s
I20260925 10:30:18.219450 31532 raft_consensus.cc:348] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.219962 31532 raft_consensus.cc:374] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:18.220117 31532 raft_consensus.cc:729] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d080209807c64ed7a46e98f193321e0c, State: Initialized, Role: FOLLOWER
I20260925 10:30:18.220834 31532 consensus_queue.cc:260] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.222745 31532 ts_tablet_manager.cc:1428] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260925 10:30:18.227041 31531 raft_consensus.cc:348] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.227428 31531 raft_consensus.cc:374] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:18.227545 31531 raft_consensus.cc:729] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d44548902174353ae21ef247fb787e2, State: Initialized, Role: FOLLOWER
I20260925 10:30:18.228097 31531 consensus_queue.cc:260] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.229168 31512 heartbeater.cc:503] Master 127.12.248.126:36175 was elected leader, sending a full tablet report...
I20260925 10:30:18.229833 31531 ts_tablet_manager.cc:1428] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Time spent starting tablet: real 0.012s user 0.008s sys 0.007s
I20260925 10:30:18.318984 31535 raft_consensus.cc:482] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:18.319465 31535 raft_consensus.cc:504] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.321293 31535 leader_election.cc:288] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d44548902174353ae21ef247fb787e2 (127.12.248.67:45481), d080209807c64ed7a46e98f193321e0c (127.12.248.65:36235)
I20260925 10:30:18.328491 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.321669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:45481 (local address 127.12.248.66:40271)
0925 10:30:18.322378 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.322404 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.322596 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.322980 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.322987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.323015 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:18.323404 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.323422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.324665 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.324670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.326362 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.326374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.327763 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.327773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.327965 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.328005 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.328099 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.328233 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":851,"threads_started":1}
I20260925 10:30:18.328619 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.322290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:40271 (local address 127.12.248.67:45481)
0925 10:30:18.322492 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.322497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.322521 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:30:18.322791 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.322795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.322857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.322938 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:18.323585 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.324476 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.326571 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.327556 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.327700 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.327752 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.328171 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.328332 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:30:18.329005 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.321909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:51085 (local address 127.12.248.65:36235)
0925 10:30:18.322168 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.322175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.322828 (+ 653us) server_negotiation.cc:407] Connection header received
0925 10:30:18.323064 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.323069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.323163 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.323233 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:18.323930 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.324870 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.326947 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.327881 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.328049 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.328126 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.328598 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.328758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 10:30:18.329294 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.321789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:36235 (local address 127.12.248.66:51085)
0925 10:30:18.322705 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:18.322722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:18.322877 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:18.323288 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:18.323294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:18.323323 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:18.323734 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.323748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.325017 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.325022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.326752 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:18.326769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.328195 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.328203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:18.328391 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.328437 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:30:18.328547 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:30:18.328997 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":562,"threads_started":1}
I20260925 10:30:18.329998 31467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bee8b5d66b84dc8a837bfd9ca391657" candidate_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d44548902174353ae21ef247fb787e2" is_pre_election: true
I20260925 10:30:18.330552 31467 raft_consensus.cc:2454] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb0e3351f284cbbbfa4dd4fd17525ac in term 0.
I20260925 10:30:18.330446 31204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bee8b5d66b84dc8a837bfd9ca391657" candidate_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d080209807c64ed7a46e98f193321e0c" is_pre_election: true
I20260925 10:30:18.331130 31204 raft_consensus.cc:2454] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb0e3351f284cbbbfa4dd4fd17525ac in term 0.
I20260925 10:30:18.331300 31268 leader_election.cc:302] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d44548902174353ae21ef247fb787e2, 7bb0e3351f284cbbbfa4dd4fd17525ac; no voters:
I20260925 10:30:18.331839 31535 raft_consensus.cc:2781] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:18.331990 31535 raft_consensus.cc:482] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:18.332121 31535 raft_consensus.cc:3037] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:18.335126 31535 raft_consensus.cc:504] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.335852 31535 leader_election.cc:288] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [CANDIDATE]: Term 1 election: Requested vote from peers 6d44548902174353ae21ef247fb787e2 (127.12.248.67:45481), d080209807c64ed7a46e98f193321e0c (127.12.248.65:36235)
I20260925 10:30:18.336416 31467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bee8b5d66b84dc8a837bfd9ca391657" candidate_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d44548902174353ae21ef247fb787e2"
I20260925 10:30:18.336454 31204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bee8b5d66b84dc8a837bfd9ca391657" candidate_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d080209807c64ed7a46e98f193321e0c"
I20260925 10:30:18.336647 31204 raft_consensus.cc:3037] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:18.336668 31467 raft_consensus.cc:3037] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:18.339001 31204 raft_consensus.cc:2454] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb0e3351f284cbbbfa4dd4fd17525ac in term 1.
I20260925 10:30:18.339121 31467 raft_consensus.cc:2454] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb0e3351f284cbbbfa4dd4fd17525ac in term 1.
I20260925 10:30:18.339660 31270 leader_election.cc:302] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb0e3351f284cbbbfa4dd4fd17525ac, d080209807c64ed7a46e98f193321e0c; no voters:
I20260925 10:30:18.340075 31535 raft_consensus.cc:2781] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:18.340668 31535 raft_consensus.cc:686] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 1 LEADER]: Becoming Leader. State: Replica: 7bb0e3351f284cbbbfa4dd4fd17525ac, State: Running, Role: LEADER
I20260925 10:30:18.341185 31535 consensus_queue.cc:237] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } }
I20260925 10:30:18.346704 31062 catalog_manager.cc:5495] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bb0e3351f284cbbbfa4dd4fd17525ac (127.12.248.66). New cstate: current_term: 1 leader_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bb0e3351f284cbbbfa4dd4fd17525ac" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 33563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 } health_report { overall_health: UNKNOWN } } }
W20260925 10:30:18.351285 31513 tablet.cc:2369] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:30:18.373342 31382 tablet.cc:2369] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:18.374715 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.369001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.12.248.66:33563)
0925 10:30:18.369205 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.369210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.369371 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:30:18.369603 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.369607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.369715 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.369851 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:18.370187 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.370857 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.372230 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.372666 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.372773 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.373574 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:18.374205 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:18.374325 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.374448 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.374514 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:30:18.383270 31204 raft_consensus.cc:1261] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 1 FOLLOWER]: Refusing update from remote peer 7bb0e3351f284cbbbfa4dd4fd17525ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:18.383270 31467 raft_consensus.cc:1261] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 1 FOLLOWER]: Refusing update from remote peer 7bb0e3351f284cbbbfa4dd4fd17525ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:18.384358 31541 consensus_queue.cc:1035] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d44548902174353ae21ef247fb787e2" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:18.384737 31535 consensus_queue.cc:1035] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d080209807c64ed7a46e98f193321e0c" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:18.412161 31061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:18.412568 31061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:18.416883 31061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7c0196a0374c43b651df75852f6749: Sending DeleteTablet for 3 replicas of tablet 4bee8b5d66b84dc8a837bfd9ca391657
I20260925 10:30:18.418040 31447 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:18 UTC) from {username='slave'} at 127.0.0.1:50318
I20260925 10:30:18.418093 31316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:18 UTC) from {username='slave'} at 127.0.0.1:35846
I20260925 10:30:18.419363 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bee8b5d66b84dc8a837bfd9ca391657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:18 UTC) from {username='slave'} at 127.0.0.1:37024
I20260925 10:30:18.420775 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31121
I20260925 10:30:18.445680 31557 tablet_replica.cc:330] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: stopping tablet replica
I20260925 10:30:18.446336 31557 raft_consensus.cc:2229] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:18.446628 31555 tablet_replica.cc:330] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: stopping tablet replica
I20260925 10:30:18.447031 31557 raft_consensus.cc:2258] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:18.447142 31555 raft_consensus.cc:2229] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:18.447830 31555 raft_consensus.cc:2258] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:18.450300 31555 ts_tablet_manager.cc:1905] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:18.451612 31556 tablet_replica.cc:330] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: stopping tablet replica
I20260925 10:30:18.452040 31556 raft_consensus.cc:2229] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:18.452576 31556 raft_consensus.cc:2258] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:18.454749 31556 ts_tablet_manager.cc:1905] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:18.455377 31557 ts_tablet_manager.cc:1905] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:18.458736 31555 ts_tablet_manager.cc:1918] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:18.459007 31555 log.cc:1196] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/4bee8b5d66b84dc8a837bfd9ca391657
I20260925 10:30:18.459658 31555 ts_tablet_manager.cc:1939] T 4bee8b5d66b84dc8a837bfd9ca391657 P 7bb0e3351f284cbbbfa4dd4fd17525ac: Deleting consensus metadata
I20260925 10:30:18.461733 31047 catalog_manager.cc:4841] TS 7bb0e3351f284cbbbfa4dd4fd17525ac (127.12.248.66:33563): tablet 4bee8b5d66b84dc8a837bfd9ca391657 (table test-table [id=2800385fd71f436b9a2c28c64136f005]) successfully deleted
I20260925 10:30:18.462122 31556 ts_tablet_manager.cc:1918] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:18.462293 31556 log.cc:1196] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/4bee8b5d66b84dc8a837bfd9ca391657
I20260925 10:30:18.462438 31557 ts_tablet_manager.cc:1918] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:18.462663 31557 log.cc:1196] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/4bee8b5d66b84dc8a837bfd9ca391657
I20260925 10:30:18.462882 31556 ts_tablet_manager.cc:1939] T 4bee8b5d66b84dc8a837bfd9ca391657 P 6d44548902174353ae21ef247fb787e2: Deleting consensus metadata
I20260925 10:30:18.463217 31557 ts_tablet_manager.cc:1939] T 4bee8b5d66b84dc8a837bfd9ca391657 P d080209807c64ed7a46e98f193321e0c: Deleting consensus metadata
I20260925 10:30:18.464717 31528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.422495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.12.248.65:36235)
0925 10:30:18.422739 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.422748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.443506 (+ 20758us) server_negotiation.cc:407] Connection header received
0925 10:30:18.443887 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.443898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.444051 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.444147 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:18.447498 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.448885 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.453588 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.454208 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.454319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.462794 (+ 8475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:18.462807 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:18.462827 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:18.463644 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:18.463712 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:18.463740 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:18.463935 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.464201 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.464331 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 10:30:18.464658 31046 catalog_manager.cc:4841] TS 6d44548902174353ae21ef247fb787e2 (127.12.248.67:45481): tablet 4bee8b5d66b84dc8a837bfd9ca391657 (table test-table [id=2800385fd71f436b9a2c28c64136f005]) successfully deleted
I20260925 10:30:18.464979 31048 catalog_manager.cc:4841] TS d080209807c64ed7a46e98f193321e0c (127.12.248.65:36235): tablet 4bee8b5d66b84dc8a837bfd9ca391657 (table test-table [id=2800385fd71f436b9a2c28c64136f005]) successfully deleted
I20260925 10:30:18.466012 31244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:18.570833 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31121
I20260925 10:30:18.595667 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31253
I20260925 10:30:18.603947 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.596669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.12.248.66:33563)
0925 10:30:18.596849 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.596854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.596872 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:30:18.596929 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.596933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.597019 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.597121 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:18.597696 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.598460 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.600464 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.600970 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.601073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.602352 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:18.602362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:18.602377 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:18.603068 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:18.603120 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:18.603141 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:18.603232 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.603398 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.603485 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:30:18.604816 31376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:18.721001 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31253
I20260925 10:30:18.740134 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31384
I20260925 10:30:18.747565 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:18.740977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.12.248.67:45481)
0925 10:30:18.741182 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:18.741188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:18.741207 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:30:18.741379 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:18.741384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:18.741529 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:18.741669 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:18.742034 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.742975 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.744411 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:18.744820 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:18.744933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:18.746125 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:18.746134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:18.746151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:18.746775 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:18.746825 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:18.746845 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:18.746938 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:18.747108 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:30:18.747194 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:30:18.748217 31507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:18.837316 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31384
I20260925 10:30:18.854224 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31031
I20260925 10:30:18.855495 31092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:18.967494 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31031
2026-09-25T10:30:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:19 dist-test-slave-f9rl krb5kdc[31565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:19 dist-test-slave-f9rl krb5kdc[31565](info): set up 2 sockets
Sep 25 10:30:19 dist-test-slave-f9rl krb5kdc[31565](info): commencing operation
krb5kdc: starting...
W20260925 10:30:20.115645 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:20 dist-test-slave-f9rl krb5kdc[31565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:30:20.385710 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43713
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:43713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:20.728885 31578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:20.729378 31578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:20.729612 31578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:20.738817 31578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:20.738960 31578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:20.739101 31578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:20.739204 31578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:20.739276 31578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:20.750820 31578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:43713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:43713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:20.752190 31578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:20.753595 31578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:20.765908 31583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:20.765934 31586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:20.765867 31584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:20.766757 31578 server_base.cc:1034] running on GCE node
I20260925 10:30:20.767518 31578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:20.769061 31578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:20.770298 31578 hybrid_clock.cc:648] HybridClock initialized: now 1790332220770249 us; error 62 us; skew 500 ppm
I20260925 10:30:20.770651 31578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:20 dist-test-slave-f9rl krb5kdc[31565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:20.773752 31578 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:30:20.775283 31578 webserver.cc:460] Webserver started at http://127.12.248.126:37821/ using document root <none> and password file <none>
I20260925 10:30:20.775970 31578 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:20.776116 31578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:20.776432 31578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:20.779515 31578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b"
format_stamp: "Formatted at 2026-09-25 10:30:20 on dist-test-slave-f9rl"
server_key: "e2cffcd037ede2b8e32727fd6ff503a4"
server_key_iv: "7705d621dc57a4dfe543ae8c43f9a0dd"
server_key_version: "encryptionkey@0"
I20260925 10:30:20.780298 31578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b"
format_stamp: "Formatted at 2026-09-25 10:30:20 on dist-test-slave-f9rl"
server_key: "e2cffcd037ede2b8e32727fd6ff503a4"
server_key_iv: "7705d621dc57a4dfe543ae8c43f9a0dd"
server_key_version: "encryptionkey@0"
I20260925 10:30:20.785746 31578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:30:20.789391 31594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:20.790884 31578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:20.791079 31578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b"
format_stamp: "Formatted at 2026-09-25 10:30:20 on dist-test-slave-f9rl"
server_key: "e2cffcd037ede2b8e32727fd6ff503a4"
server_key_iv: "7705d621dc57a4dfe543ae8c43f9a0dd"
server_key_version: "encryptionkey@0"
I20260925 10:30:20.791296 31578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:20.804514 31578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:20.808362 31578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:20.808676 31578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:20.827519 31578 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:43713
I20260925 10:30:20.827530 31645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:43713 every 8 connection(s)
I20260925 10:30:20.828984 31578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:20.831480 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31578
I20260925 10:30:20.831688 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:20.832304 13281 external_mini_cluster.cc:1428] Setting key c8e5d6fa1dc7c892c90d0dd745df298e
I20260925 10:30:20.833325 31646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 10:30:20 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:20.848101 31646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: Bootstrap starting.
I20260925 10:30:20.851578 31646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:20.852880 31646 log.cc:824] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:20.856050 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:20.834216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.12.248.126:43713)
0925 10:30:20.835005 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:20.835017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:20.835057 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:30:20.836689 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:20.836710 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:20.837117 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:20.837551 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:20.839339 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:20.840686 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:20.842684 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:20.843346 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:20.843482 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:20.848374 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:20.848382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:20.848397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:20.848437 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:20.851647 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:20.852539 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:20.852546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:20.852552 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:20.852741 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:20.853180 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:20.853184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:20.853186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:20.853940 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:20.854313 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:20.854589 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 10:30:20.854957 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":381,"threads_started":1}
I20260925 10:30:20.856288 31646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: No bootstrap required, opened a new log
I20260925 10:30:20.864322 31646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } }
I20260925 10:30:20.864749 31646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:20.864861 31646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd7fa2fbae64fcdb0f1b1de5f07890b, State: Initialized, Role: FOLLOWER
I20260925 10:30:20.865656 31646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } }
I20260925 10:30:20.865948 31646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:20.866142 31646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:20.866359 31646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:20.868592 31646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } }
I20260925 10:30:20.869139 31646 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd7fa2fbae64fcdb0f1b1de5f07890b; no voters:
I20260925 10:30:20.869849 31646 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:20.870066 31651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:20.871438 31651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [term 1 LEADER]: Becoming Leader. State: Replica: 3dd7fa2fbae64fcdb0f1b1de5f07890b, State: Running, Role: LEADER
I20260925 10:30:20.871948 31651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } }
I20260925 10:30:20.872424 31646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:20.874724 31653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd7fa2fbae64fcdb0f1b1de5f07890b. Latest consensus state: current_term: 1 leader_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } } }
I20260925 10:30:20.875216 31653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:20.875175 31652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd7fa2fbae64fcdb0f1b1de5f07890b" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 43713 } } }
I20260925 10:30:20.875558 31652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:20.877236 31660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:20.882712 31660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:20.891317 31660 catalog_manager.cc:1347] Generated new cluster ID: 0525480531b64eeea78ec85b8ef5e44d
I20260925 10:30:20.891443 31660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:20.904230 31660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:20.906194 31660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:20.921226 31660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: Generated new TSK 0
I20260925 10:30:20.922457 31660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:30:20.993644 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:21.340416 31672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:21.340727 31672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:21.340912 31672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:21.348414 31672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:21.348555 31672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:21.348753 31672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:21.359284 31672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:21.360706 31672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:21.362177 31672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:21.375957 31677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:21.375967 31680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:21.375959 31678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:21.376637 31672 server_base.cc:1034] running on GCE node
I20260925 10:30:21.377370 31672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:21.378443 31672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:21.379722 31672 hybrid_clock.cc:648] HybridClock initialized: now 1790332221379678 us; error 39 us; skew 500 ppm
I20260925 10:30:21.380103 31672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:21 dist-test-slave-f9rl krb5kdc[31565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:21.383354 31672 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:30:21.384999 31672 webserver.cc:460] Webserver started at http://127.12.248.65:39859/ using document root <none> and password file <none>
I20260925 10:30:21.385811 31672 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:21.385965 31672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:21.386296 31672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:21.389243 31672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "8f520c94d3364c7bbf76b7222d873a97"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "6f88308a526d938edb188202a73bf7e7"
server_key_iv: "0a93763ea62ed2cc017159986c47692e"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.390064 31672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "8f520c94d3364c7bbf76b7222d873a97"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "6f88308a526d938edb188202a73bf7e7"
server_key_iv: "0a93763ea62ed2cc017159986c47692e"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.395546 31672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 10:30:21.399471 31687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:21.400976 31672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:30:21.401162 31672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "8f520c94d3364c7bbf76b7222d873a97"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "6f88308a526d938edb188202a73bf7e7"
server_key_iv: "0a93763ea62ed2cc017159986c47692e"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.401373 31672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:21.422391 31672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:21.427027 31672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:21.427345 31672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:21.428337 31672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:21.429754 31672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:21.429834 31672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:21.429996 31672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:21.430058 31672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:21.462889 31672 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:39355
I20260925 10:30:21.462937 31800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:39355 every 8 connection(s)
I20260925 10:30:21.464247 31672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:21.472990 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31672
I20260925 10:30:21.473183 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:21.473748 13281 external_mini_cluster.cc:1428] Setting key 45a21aa07847b9a4f132a8288d11ddcd
Sep 25 10:30:21 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:21.484647 31803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:21.466830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:52015 (local address 127.12.248.126:43713)
0925 10:30:21.467349 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:21.467354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:21.468186 (+ 832us) server_negotiation.cc:407] Connection header received
0925 10:30:21.470149 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:21.470155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:21.470254 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:21.470373 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:21.471846 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:21.472685 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:21.474632 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:21.475046 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:21.475226 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:21.478897 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:21.478902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:21.478905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:21.478936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:21.481573 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:21.482299 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:21.482304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:21.482309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:21.482429 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:21.482873 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:21.482878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:21.482882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:21.483156 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:21.483312 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:21.484179 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 10:30:21.484375 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":309,"threads_started":1}
I20260925 10:30:21.486316 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:21.467302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43713 (local address 127.12.248.65:52015)
0925 10:30:21.468012 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:21.468049 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:21.469855 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:21.470609 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:21.470617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:21.471119 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:21.471674 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:21.471691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:21.472854 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:21.472859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:21.474480 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:21.474492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:21.475285 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:21.475295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:21.475567 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:21.476298 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:21.476325 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:21.478664 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:21.481716 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:21.481725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:21.481743 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:21.482171 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:21.482540 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:21.482545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:21.482548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:21.482752 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:21.483285 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:21.483293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:21.483745 (+ 452us) client_negotiation.cc:769] Sending connection context
0925 10:30:21.484047 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 10:30:21.484664 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
I20260925 10:30:21.487980 31801 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43713
I20260925 10:30:21.488297 31801 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:21.489018 31801 heartbeater.cc:511] Master 127.12.248.126:43713 requested a full tablet report, sending...
I20260925 10:30:21.491245 31611 ts_manager.cc:194] Registered new tserver with Master: 8f520c94d3364c7bbf76b7222d873a97 (127.12.248.65:39355)
I20260925 10:30:21.492718 31611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:52015
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:30:21.533144 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:21.864274 31807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:21.864638 31807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:21.864817 31807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:21.872471 31807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:21.872665 31807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:21.872859 31807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:21.883302 31807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:21.884752 31807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:21.886205 31807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:21.900038 31812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:21.900166 31813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:21.900169 31815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:21.901759 31807 server_base.cc:1034] running on GCE node
I20260925 10:30:21.902372 31807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:21.903388 31807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:21.904662 31807 hybrid_clock.cc:648] HybridClock initialized: now 1790332221904588 us; error 60 us; skew 500 ppm
I20260925 10:30:21.905002 31807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:21 dist-test-slave-f9rl krb5kdc[31565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:21.913408 31807 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:30:21.915108 31807 webserver.cc:460] Webserver started at http://127.12.248.66:41063/ using document root <none> and password file <none>
I20260925 10:30:21.915861 31807 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:21.916011 31807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:21.916342 31807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:21.919437 31807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "cee09b98ffc24525aeba089db63e6e16"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "48db69ab46b16f6be490040009b108ec"
server_key_iv: "0d95cf61b8b7d45c56849bd956f9413d"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.920225 31807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "cee09b98ffc24525aeba089db63e6e16"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "48db69ab46b16f6be490040009b108ec"
server_key_iv: "0d95cf61b8b7d45c56849bd956f9413d"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.925560 31807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 10:30:21.929033 31822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:21.930415 31807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:30:21.930596 31807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "cee09b98ffc24525aeba089db63e6e16"
format_stamp: "Formatted at 2026-09-25 10:30:21 on dist-test-slave-f9rl"
server_key: "48db69ab46b16f6be490040009b108ec"
server_key_iv: "0d95cf61b8b7d45c56849bd956f9413d"
server_key_version: "encryptionkey@0"
I20260925 10:30:21.930814 31807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:21.949966 31807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:21.954454 31807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:21.954773 31807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:21.955780 31807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:21.957193 31807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:21.957309 31807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:21.957501 31807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:21.957576 31807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 10:30:21.991385 31807 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:41033
I20260925 10:30:21.991420 31935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:41033 every 8 connection(s)
I20260925 10:30:21.992805 31807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:21.998466 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31807
I20260925 10:30:21.998682 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:21.999152 13281 external_mini_cluster.cc:1428] Setting key 62f143816c9b4541ceba2e2a239b22c6
Sep 25 10:30:22 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:22.019372 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:21.995523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:40827 (local address 127.12.248.126:43713)
0925 10:30:21.995931 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:21.995935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:21.997077 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 10:30:21.999141 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:21.999147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:21.999237 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:21.999346 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:22.001547 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.002648 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.005284 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.005846 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.005967 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.010777 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:22.010782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:22.010785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:22.010818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:22.015230 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.016139 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.016148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.016155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.016348 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.016834 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.016843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.016850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.017360 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:22.017638 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.018493 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.018823 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":197,"threads_started":1}
I20260925 10:30:22.019979 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:21.996195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43713 (local address 127.12.248.66:40827)
0925 10:30:21.996905 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:21.996942 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:21.998869 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:21.999639 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:21.999653 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.000411 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:22.001224 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.001255 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.002845 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.002854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.005128 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.005147 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.006099 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.006109 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.006457 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.007242 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:22.007281 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:22.010505 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:22.015446 (+ 4941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:22.015454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:22.015469 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:22.015961 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:22.016482 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:22.016485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:22.016488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:22.016705 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:22.017641 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:22.017647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:22.018050 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.018341 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.018771 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
I20260925 10:30:22.021677 31936 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43713
I20260925 10:30:22.022037 31936 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:22.023118 31936 heartbeater.cc:511] Master 127.12.248.126:43713 requested a full tablet report, sending...
I20260925 10:30:22.025012 31611 ts_manager.cc:194] Registered new tserver with Master: cee09b98ffc24525aeba089db63e6e16 (127.12.248.66:41033)
I20260925 10:30:22.025972 31611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:40827
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:30:22.066450 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:22.402006 31942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:22.402345 31942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:22.402518 31942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:22.410115 31942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:22.410267 31942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:22.410439 31942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:22.420709 31942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:43713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:22.422245 31942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:22.423687 31942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:22.440516 31947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:22.440516 31948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:22.440541 31951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:22.441504 31942 server_base.cc:1034] running on GCE node
I20260925 10:30:22.442121 31942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:22.443120 31942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:22.444331 31942 hybrid_clock.cc:648] HybridClock initialized: now 1790332222444288 us; error 37 us; skew 500 ppm
I20260925 10:30:22.444636 31942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:22 dist-test-slave-f9rl krb5kdc[31565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:22.447775 31942 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:30:22.449175 31942 webserver.cc:460] Webserver started at http://127.12.248.67:46361/ using document root <none> and password file <none>
I20260925 10:30:22.449900 31942 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:22.450006 31942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:22.450350 31942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:22.453238 31942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "24a9f495f4d148da9731bb8a7440efc1"
format_stamp: "Formatted at 2026-09-25 10:30:22 on dist-test-slave-f9rl"
server_key: "a4d08a0e0d00bb8642a57d3cecbad9ce"
server_key_iv: "ff757b71d5c979a71c68458956c914ee"
server_key_version: "encryptionkey@0"
I20260925 10:30:22.454026 31942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "24a9f495f4d148da9731bb8a7440efc1"
format_stamp: "Formatted at 2026-09-25 10:30:22 on dist-test-slave-f9rl"
server_key: "a4d08a0e0d00bb8642a57d3cecbad9ce"
server_key_iv: "ff757b71d5c979a71c68458956c914ee"
server_key_version: "encryptionkey@0"
I20260925 10:30:22.459275 31942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:22.463234 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:22.464864 31942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:30:22.465034 31942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "24a9f495f4d148da9731bb8a7440efc1"
format_stamp: "Formatted at 2026-09-25 10:30:22 on dist-test-slave-f9rl"
server_key: "a4d08a0e0d00bb8642a57d3cecbad9ce"
server_key_iv: "ff757b71d5c979a71c68458956c914ee"
server_key_version: "encryptionkey@0"
I20260925 10:30:22.465296 31942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:22.496136 31801 heartbeater.cc:503] Master 127.12.248.126:43713 was elected leader, sending a full tablet report...
I20260925 10:30:22.507771 31942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:22.511756 31942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:22.512066 31942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:22.513041 31942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:22.514513 31942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:22.514586 31942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:22.514770 31942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:22.514832 31942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:22.548550 31942 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:40629
I20260925 10:30:22.548646 32070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:40629 every 8 connection(s)
I20260925 10:30:22.549980 31942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:22.552253 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 31942
I20260925 10:30:22.552527 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:22.553032 13281 external_mini_cluster.cc:1428] Setting key 8efaa024272a91ac688f5716c690f3e4
Sep 25 10:30:22 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:22.571851 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.553261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:42559 (local address 127.12.248.126:43713)
0925 10:30:22.553792 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.553797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.554997 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 10:30:22.557157 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.557162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.557257 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.557382 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:22.558929 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.559721 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.561370 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.561842 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.561975 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.565705 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:22.565710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:22.565714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:22.565746 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:22.568975 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.569725 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.569728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.569731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.569870 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.570244 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.570247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.570249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.570531 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:22.570690 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.571353 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.571510 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":254,"threads_started":1}
I20260925 10:30:22.572835 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.553942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:43713 (local address 127.12.248.67:42559)
0925 10:30:22.554825 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.554883 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.556889 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.557657 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.557665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.558169 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:22.558757 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.558774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.559873 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.559878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.561215 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.561227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.562080 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.562087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.562333 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.563046 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:22.563068 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:22.565478 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:22.569093 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:22.569100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:22.569112 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:22.569612 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:22.569950 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:22.569953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:22.569955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:22.570122 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:22.570673 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:22.570680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:22.571006 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.571249 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.571613 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260925 10:30:22.574146 32071 heartbeater.cc:347] Connected to a master server at 127.12.248.126:43713
I20260925 10:30:22.574460 32071 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:22.575201 32071 heartbeater.cc:511] Master 127.12.248.126:43713 requested a full tablet report, sending...
I20260925 10:30:22.576620 31611 ts_manager.cc:194] Registered new tserver with Master: 24a9f495f4d148da9731bb8a7440efc1 (127.12.248.67:40629)
I20260925 10:30:22.577515 31611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:42559
I20260925 10:30:22.580714 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:22.598968 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.584864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.12.248.126:43713)
0925 10:30:22.585063 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.585072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.585483 (+ 411us) server_negotiation.cc:407] Connection header received
0925 10:30:22.585765 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.585769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.585835 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.585939 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:22.587391 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.588127 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.589615 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.590058 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.590207 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.593180 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:22.593185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:22.593189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:22.593224 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:22.595923 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.596884 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.596888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.596890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.597011 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.597504 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.597508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.597510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.597829 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:22.598042 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.598420 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.598564 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:30:22.603603 31611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:22.610890 31611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:22.646577 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.624960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.12.248.66:41033)
0925 10:30:22.636730 (+ 11770us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.636742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.638012 (+ 1270us) server_negotiation.cc:407] Connection header received
0925 10:30:22.638139 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.638146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.638397 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.638597 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:22.639554 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641315 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.643803 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.645159 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.645295 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.645540 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.645997 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.646167 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11546,"thread_start_us":11402,"threads_started":1}
I20260925 10:30:22.646644 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.624404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41033 (local address 127.0.0.1:46004)
0925 10:30:22.637737 (+ 13333us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.637759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.637896 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.638636 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.638669 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.638725 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:22.639265 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.639279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641503 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.641509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.643637 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.643650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.645644 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.645648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.645780 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.645812 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.645954 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.646167 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13085,"mutex_wait_us":116,"spinlock_wait_cycles":1280,"thread_start_us":12947,"threads_started":1}
I20260925 10:30:22.647436 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.624950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:40629 (local address 127.0.0.1:49132)
0925 10:30:22.638567 (+ 13617us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.638585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.638703 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.639260 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.639267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.639298 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:22.639748 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.639766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641936 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.641942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.644354 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.644368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.646543 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.646562 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.646738 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.646770 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.646859 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.647010 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13474,"mutex_wait_us":37,"thread_start_us":9529,"threads_started":1}
I20260925 10:30:22.647655 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.625738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.12.248.67:40629)
0925 10:30:22.636185 (+ 10447us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.636198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.638711 (+ 2513us) server_negotiation.cc:407] Connection header received
0925 10:30:22.638823 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.638833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.639056 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.639253 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:22.639936 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641794 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.644511 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.645987 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.646115 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.646383 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.647157 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.647331 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10208,"thread_start_us":10003,"threads_started":1}
I20260925 10:30:22.647861 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.624029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:39355 (local address 127.0.0.1:37540)
0925 10:30:22.635028 (+ 10999us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.635097 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.635382 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.638912 (+ 3530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.638916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.638948 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:22.639548 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.639563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641503 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.641509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.644281 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.644298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.647005 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.647011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.647205 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.647235 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.647329 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.647485 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10646,"mutex_wait_us":138,"thread_start_us":14234,"threads_started":1}
I20260925 10:30:22.648654 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.624360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.12.248.65:39355)
0925 10:30:22.637304 (+ 12944us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.637316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.637354 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:30:22.637497 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.637506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.637812 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.638928 (+ 1116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:22.639747 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.641315 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.644490 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.646516 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.646651 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.646865 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.648179 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.648335 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12767,"thread_start_us":12565,"threads_started":1}
I20260925 10:30:22.651194 31870 tablet_service.cc:1467] Processing CreateTablet for tablet 2825546dc0214623827fda708248edbf (DEFAULT_TABLE table=test-table [id=55675cfcd90c43e69e6311b8323a8519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:22.651801 32006 tablet_service.cc:1467] Processing CreateTablet for tablet 2825546dc0214623827fda708248edbf (DEFAULT_TABLE table=test-table [id=55675cfcd90c43e69e6311b8323a8519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:22.652622 31870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2825546dc0214623827fda708248edbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:22.652263 31735 tablet_service.cc:1467] Processing CreateTablet for tablet 2825546dc0214623827fda708248edbf (DEFAULT_TABLE table=test-table [id=55675cfcd90c43e69e6311b8323a8519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:22.653344 32006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2825546dc0214623827fda708248edbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:22.653697 31735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2825546dc0214623827fda708248edbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:22.664776 32091 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Bootstrap starting.
I20260925 10:30:22.668980 32092 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Bootstrap starting.
I20260925 10:30:22.669114 32091 tablet_bootstrap.cc:654] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:22.669659 32093 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Bootstrap starting.
I20260925 10:30:22.670579 32091 log.cc:824] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:22.672971 32092 tablet_bootstrap.cc:654] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:22.673374 32093 tablet_bootstrap.cc:654] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:22.674293 32091 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: No bootstrap required, opened a new log
I20260925 10:30:22.674371 32092 log.cc:824] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:22.674592 32093 log.cc:824] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:22.674618 32091 ts_tablet_manager.cc:1397] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 10:30:22.677659 32092 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: No bootstrap required, opened a new log
I20260925 10:30:22.677759 32093 tablet_bootstrap.cc:492] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: No bootstrap required, opened a new log
I20260925 10:30:22.677923 32092 ts_tablet_manager.cc:1397] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 10:30:22.677964 32093 ts_tablet_manager.cc:1397] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 10:30:22.683486 32091 raft_consensus.cc:348] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.683883 32091 raft_consensus.cc:374] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:22.683990 32091 raft_consensus.cc:729] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee09b98ffc24525aeba089db63e6e16, State: Initialized, Role: FOLLOWER
I20260925 10:30:22.684629 32091 consensus_queue.cc:260] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.685792 31936 heartbeater.cc:503] Master 127.12.248.126:43713 was elected leader, sending a full tablet report...
I20260925 10:30:22.686493 32091 ts_tablet_manager.cc:1428] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260925 10:30:22.688923 32092 raft_consensus.cc:348] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.688925 32093 raft_consensus.cc:348] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.689356 32092 raft_consensus.cc:374] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:22.689356 32093 raft_consensus.cc:374] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:22.689482 32093 raft_consensus.cc:729] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f520c94d3364c7bbf76b7222d873a97, State: Initialized, Role: FOLLOWER
I20260925 10:30:22.689515 32092 raft_consensus.cc:729] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a9f495f4d148da9731bb8a7440efc1, State: Initialized, Role: FOLLOWER
I20260925 10:30:22.690212 32092 consensus_queue.cc:260] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.690212 32093 consensus_queue.cc:260] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.692095 32071 heartbeater.cc:503] Master 127.12.248.126:43713 was elected leader, sending a full tablet report...
I20260925 10:30:22.692238 32092 ts_tablet_manager.cc:1428] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 10:30:22.692652 32093 ts_tablet_manager.cc:1428] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260925 10:30:22.716776 31802 tablet.cc:2369] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:30:22.745018 31937 tablet.cc:2369] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:22.748605 32098 raft_consensus.cc:482] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:22.748910 32098 raft_consensus.cc:504] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.750675 32098 leader_election.cc:288] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee09b98ffc24525aeba089db63e6e16 (127.12.248.66:41033), 24a9f495f4d148da9731bb8a7440efc1 (127.12.248.67:40629)
I20260925 10:30:22.757808 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.751236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:43439 (local address 127.12.248.66:41033)
0925 10:30:22.751500 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.751508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.752220 (+ 712us) server_negotiation.cc:407] Connection header received
0925 10:30:22.752394 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.752399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.752477 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.752588 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:22.753219 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.754084 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.755951 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.756773 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.756890 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.757000 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.757364 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.757538 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:30:22.757807 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.751335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:57643 (local address 127.12.248.67:40629)
0925 10:30:22.751547 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.751553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.752403 (+ 850us) server_negotiation.cc:407] Connection header received
0925 10:30:22.752597 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.752601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.752678 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.752780 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:22.753397 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.754184 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.756063 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.756820 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.757005 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.757078 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.757405 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.757550 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:30:22.757961 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.751334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:40629 (local address 127.12.248.65:57643)
0925 10:30:22.752263 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.752287 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.752439 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.752815 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.752819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.752841 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:22.753234 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.753245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.754326 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.754332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.755907 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.755922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.757099 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.757104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.757229 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.757251 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.757361 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.757511 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":25,"thread_start_us":549,"threads_started":1}
I20260925 10:30:22.758224 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.750977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:41033 (local address 127.12.248.65:43439)
0925 10:30:22.752098 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:22.752114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:22.752251 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:22.752613 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:22.752618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:22.752641 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:22.753066 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.753077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.754222 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.754230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.755786 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:22.755801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.756993 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.757000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:22.757180 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.757232 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:30:22.757306 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:30:22.757468 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":21,"thread_start_us":760,"threads_started":1}
I20260925 10:30:22.759063 32026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2825546dc0214623827fda708248edbf" candidate_uuid: "8f520c94d3364c7bbf76b7222d873a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a9f495f4d148da9731bb8a7440efc1" is_pre_election: true
I20260925 10:30:22.759115 31890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2825546dc0214623827fda708248edbf" candidate_uuid: "8f520c94d3364c7bbf76b7222d873a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee09b98ffc24525aeba089db63e6e16" is_pre_election: true
I20260925 10:30:22.759562 32026 raft_consensus.cc:2454] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f520c94d3364c7bbf76b7222d873a97 in term 0.
I20260925 10:30:22.759581 31890 raft_consensus.cc:2454] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f520c94d3364c7bbf76b7222d873a97 in term 0.
I20260925 10:30:22.760228 31691 leader_election.cc:302] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a9f495f4d148da9731bb8a7440efc1, 8f520c94d3364c7bbf76b7222d873a97; no voters:
I20260925 10:30:22.760732 32098 raft_consensus.cc:2781] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:22.760867 32098 raft_consensus.cc:482] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:22.760949 32098 raft_consensus.cc:3037] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:22.763356 32098 raft_consensus.cc:504] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.763942 32098 leader_election.cc:288] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [CANDIDATE]: Term 1 election: Requested vote from peers cee09b98ffc24525aeba089db63e6e16 (127.12.248.66:41033), 24a9f495f4d148da9731bb8a7440efc1 (127.12.248.67:40629)
I20260925 10:30:22.764415 31890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2825546dc0214623827fda708248edbf" candidate_uuid: "8f520c94d3364c7bbf76b7222d873a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee09b98ffc24525aeba089db63e6e16"
I20260925 10:30:22.764503 32026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2825546dc0214623827fda708248edbf" candidate_uuid: "8f520c94d3364c7bbf76b7222d873a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a9f495f4d148da9731bb8a7440efc1"
I20260925 10:30:22.764680 32026 raft_consensus.cc:3037] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:22.764636 31890 raft_consensus.cc:3037] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:22.767966 31890 raft_consensus.cc:2454] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f520c94d3364c7bbf76b7222d873a97 in term 1.
I20260925 10:30:22.767966 32026 raft_consensus.cc:2454] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f520c94d3364c7bbf76b7222d873a97 in term 1.
I20260925 10:30:22.768414 31690 leader_election.cc:302] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f520c94d3364c7bbf76b7222d873a97, cee09b98ffc24525aeba089db63e6e16; no voters:
I20260925 10:30:22.768712 32098 raft_consensus.cc:2781] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:22.769165 32098 raft_consensus.cc:686] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 1 LEADER]: Becoming Leader. State: Replica: 8f520c94d3364c7bbf76b7222d873a97, State: Running, Role: LEADER
I20260925 10:30:22.769728 32098 consensus_queue.cc:237] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } }
I20260925 10:30:22.774636 31611 catalog_manager.cc:5495] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f520c94d3364c7bbf76b7222d873a97 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "8f520c94d3364c7bbf76b7222d873a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f520c94d3364c7bbf76b7222d873a97" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 39355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 } health_report { overall_health: UNKNOWN } } }
W20260925 10:30:22.802377 32072 tablet.cc:2369] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:22.805058 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.799202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.12.248.65:39355)
0925 10:30:22.799414 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.799421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.799448 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:30:22.799614 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.799619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.799699 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.799876 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:22.800324 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.801092 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.802555 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.803232 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.803396 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.803836 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:22.804366 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:22.804521 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.804666 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.804710 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:30:22.813225 31890 raft_consensus.cc:1261] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 1 FOLLOWER]: Refusing update from remote peer 8f520c94d3364c7bbf76b7222d873a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:22.813213 32026 raft_consensus.cc:1261] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 1 FOLLOWER]: Refusing update from remote peer 8f520c94d3364c7bbf76b7222d873a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:22.814261 32102 consensus_queue.cc:1035] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee09b98ffc24525aeba089db63e6e16" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:22.814690 32098 consensus_queue.cc:1035] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a9f495f4d148da9731bb8a7440efc1" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:22.844372 31611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:22.844678 31611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:22.849198 31611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dd7fa2fbae64fcdb0f1b1de5f07890b: Sending DeleteTablet for 3 replicas of tablet 2825546dc0214623827fda708248edbf
I20260925 10:30:22.850564 31870 tablet_service.cc:1514] Processing DeleteTablet for tablet 2825546dc0214623827fda708248edbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:22 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:46004
I20260925 10:30:22.850914 31735 tablet_service.cc:1514] Processing DeleteTablet for tablet 2825546dc0214623827fda708248edbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:22 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:37540
I20260925 10:30:22.850950 32006 tablet_service.cc:1514] Processing DeleteTablet for tablet 2825546dc0214623827fda708248edbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:22 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:49132
I20260925 10:30:22.852350 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31672
I20260925 10:30:22.863137 32115 tablet_replica.cc:330] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: stopping tablet replica
I20260925 10:30:22.863827 32116 tablet_replica.cc:330] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: stopping tablet replica
I20260925 10:30:22.864511 32116 raft_consensus.cc:2229] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:22.865048 32115 raft_consensus.cc:2229] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:22.865865 32116 raft_consensus.cc:2258] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:22.866479 32115 raft_consensus.cc:2258] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:22.869232 32116 ts_tablet_manager.cc:1905] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:22.869731 32115 ts_tablet_manager.cc:1905] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:22.870075 32117 tablet_replica.cc:330] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: stopping tablet replica
I20260925 10:30:22.876710 32117 raft_consensus.cc:2229] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:22.877514 32117 raft_consensus.cc:2258] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 10:30:22 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:30:22.879057 32116 ts_tablet_manager.cc:1918] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:22.879316 32116 log.cc:1196] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/2825546dc0214623827fda708248edbf
I20260925 10:30:22.879801 32117 ts_tablet_manager.cc:1905] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:22.879956 32116 ts_tablet_manager.cc:1939] T 2825546dc0214623827fda708248edbf P 8f520c94d3364c7bbf76b7222d873a97: Deleting consensus metadata
I20260925 10:30:22.882745 31595 catalog_manager.cc:4841] TS 8f520c94d3364c7bbf76b7222d873a97 (127.12.248.65:39355): tablet 2825546dc0214623827fda708248edbf (table test-table [id=55675cfcd90c43e69e6311b8323a8519]) successfully deleted
I20260925 10:30:22.884295 32115 ts_tablet_manager.cc:1918] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:22.884591 32115 log.cc:1196] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/2825546dc0214623827fda708248edbf
I20260925 10:30:22.885278 32115 ts_tablet_manager.cc:1939] T 2825546dc0214623827fda708248edbf P cee09b98ffc24525aeba089db63e6e16: Deleting consensus metadata
I20260925 10:30:22.887198 32117 ts_tablet_manager.cc:1918] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:22.887452 32117 log.cc:1196] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/2825546dc0214623827fda708248edbf
I20260925 10:30:22.887472 31597 catalog_manager.cc:4841] TS cee09b98ffc24525aeba089db63e6e16 (127.12.248.66:41033): tablet 2825546dc0214623827fda708248edbf (table test-table [id=55675cfcd90c43e69e6311b8323a8519]) successfully deleted
I20260925 10:30:22.887992 32117 ts_tablet_manager.cc:1939] T 2825546dc0214623827fda708248edbf P 24a9f495f4d148da9731bb8a7440efc1: Deleting consensus metadata
I20260925 10:30:22.889128 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:22.854272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.12.248.65:39355)
0925 10:30:22.854514 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:22.854523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:22.866650 (+ 12127us) server_negotiation.cc:407] Connection header received
0925 10:30:22.867364 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:22.867377 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:22.867470 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:22.867550 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:22.869700 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.870936 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.872961 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:22.873715 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:22.873826 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:22.880328 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:22.880344 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:22.880362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:22.880416 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:22.884560 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.885681 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.885691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.885694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.885900 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:22.886432 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:22.886437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:22.886440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:22.886939 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:22.887413 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:22.888379 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 10:30:22.888651 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:30:22.889918 31796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:22.889902 31598 catalog_manager.cc:4841] TS 24a9f495f4d148da9731bb8a7440efc1 (127.12.248.67:40629): tablet 2825546dc0214623827fda708248edbf (table test-table [id=55675cfcd90c43e69e6311b8323a8519]) successfully deleted
I20260925 10:30:22.990496 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31672
I20260925 10:30:23.011482 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31807
Sep 25 10:30:23 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:30:23.027105 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:23.012370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.12.248.66:41033)
0925 10:30:23.012549 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:23.012554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:23.012572 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:30:23.012718 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:23.012722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:23.012783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:23.012874 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:23.014267 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.015076 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.016291 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.016677 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.016771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:23.021230 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:23.021244 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:23.021259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:23.021295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:23.024368 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:23.025082 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:23.025089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:23.025091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:23.025212 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:23.025651 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:23.025655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:23.025657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:23.026048 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:23.026300 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:23.026496 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:30:23.026733 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:30:23.027853 31931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:23.125510 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31807
I20260925 10:30:23.143066 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31942
Sep 25 10:30:23 dist-test-slave-f9rl krb5kdc[31565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:30:23.159448 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:23.143942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.12.248.67:40629)
0925 10:30:23.144117 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:23.144121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:23.144228 (+ 107us) server_negotiation.cc:407] Connection header received
0925 10:30:23.144384 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:23.144388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:23.144454 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:23.144542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:23.146110 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.146843 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.148352 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.148792 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.148889 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:23.153584 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:23.153602 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:23.153620 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:23.153667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:23.156657 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:23.157336 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:23.157342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:23.157344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:23.157775 (+ 431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:23.158130 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:23.158133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:23.158135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:23.158483 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:23.158766 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:23.159038 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:30:23.159204 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:30:23.160180 32066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:23.250384 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31942
I20260925 10:30:23.268114 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 31578
I20260925 10:30:23.269366 31641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:23.384145 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 31578
2026-09-25T10:30:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4418 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:23Z Disabled control of system clock
I20260925 10:30:23.463735 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:45745
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:23.801491 32126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:23.801800 32126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:23.801883 32126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:23.809278 32126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:23.809388 32126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:23.809427 32126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:23.809552 32126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:23.809631 32126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:23.819381 32126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:45745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:23.820725 32126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:23.822561 32126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:23.835099 32131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:23.835063 32134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:23.835049 32132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:23.836028 32126 server_base.cc:1034] running on GCE node
I20260925 10:30:23.836853 32126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:23.838375 32126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:23.839691 32126 hybrid_clock.cc:648] HybridClock initialized: now 1790332223839558 us; error 62 us; skew 500 ppm
I20260925 10:30:23.840078 32126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:23.842736 32126 webserver.cc:460] Webserver started at http://127.12.248.126:41137/ using document root <none> and password file <none>
I20260925 10:30:23.843494 32126 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:23.843636 32126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:23.843984 32126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:23.847198 32126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "95515921571f4a4ab818b047b3bb25e2"
format_stamp: "Formatted at 2026-09-25 10:30:23 on dist-test-slave-f9rl"
server_key: "003b54fff619a168fe56a8b038a00978"
server_key_iv: "277247ac7548e65aac694ee4c522981c"
server_key_version: "encryptionkey@0"
I20260925 10:30:23.848052 32126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "95515921571f4a4ab818b047b3bb25e2"
format_stamp: "Formatted at 2026-09-25 10:30:23 on dist-test-slave-f9rl"
server_key: "003b54fff619a168fe56a8b038a00978"
server_key_iv: "277247ac7548e65aac694ee4c522981c"
server_key_version: "encryptionkey@0"
I20260925 10:30:23.853746 32126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 10:30:23.857785 32140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:23.859299 32126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:23.859519 32126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "95515921571f4a4ab818b047b3bb25e2"
format_stamp: "Formatted at 2026-09-25 10:30:23 on dist-test-slave-f9rl"
server_key: "003b54fff619a168fe56a8b038a00978"
server_key_iv: "277247ac7548e65aac694ee4c522981c"
server_key_version: "encryptionkey@0"
I20260925 10:30:23.859824 32126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:23.907407 32126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:23.908375 32126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:23.908706 32126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:23.928370 32126 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:45745
I20260925 10:30:23.928398 32191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:45745 every 8 connection(s)
I20260925 10:30:23.929839 32126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:23.934060 32192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:23.938767 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32126
I20260925 10:30:23.938959 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:23.939435 13281 external_mini_cluster.cc:1428] Setting key 2a117ed5dc338b42d47c829a128a2352
I20260925 10:30:23.948514 32192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: Bootstrap starting.
I20260925 10:30:23.952948 32192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:23.954391 32192 log.cc:824] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:23.957238 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:23.941654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.12.248.126:45745)
0925 10:30:23.942859 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:23.942881 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:23.942939 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:30:23.946325 (+ 3386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:23.946362 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:23.947065 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:23.947580 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:23.948204 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.950437 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.951885 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:23.953036 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:23.953221 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:23.954404 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:23.954419 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:23.954439 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:23.955229 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:23.955315 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:23.955339 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:23.955434 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:23.955671 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:30:23.955993 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":380,"threads_started":1}
I20260925 10:30:23.957903 32192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: No bootstrap required, opened a new log
I20260925 10:30:23.965973 32192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } }
I20260925 10:30:23.966393 32192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:23.966516 32192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95515921571f4a4ab818b047b3bb25e2, State: Initialized, Role: FOLLOWER
I20260925 10:30:23.967072 32192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } }
I20260925 10:30:23.967330 32192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:23.967478 32192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:23.967670 32192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:23.970084 32192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } }
I20260925 10:30:23.970607 32192 leader_election.cc:302] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95515921571f4a4ab818b047b3bb25e2; no voters:
I20260925 10:30:23.971212 32192 leader_election.cc:288] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:23.971359 32197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:23.972080 32197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [term 1 LEADER]: Becoming Leader. State: Replica: 95515921571f4a4ab818b047b3bb25e2, State: Running, Role: LEADER
I20260925 10:30:23.972580 32197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } }
I20260925 10:30:23.973317 32192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:23.975342 32199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95515921571f4a4ab818b047b3bb25e2. Latest consensus state: current_term: 1 leader_uuid: "95515921571f4a4ab818b047b3bb25e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } } }
I20260925 10:30:23.975721 32199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:23.976200 32198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95515921571f4a4ab818b047b3bb25e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95515921571f4a4ab818b047b3bb25e2" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45745 } } }
I20260925 10:30:23.976615 32198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:23.978480 32202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:23.984383 32202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:23.993089 32202 catalog_manager.cc:1347] Generated new cluster ID: 9cc569cb57c34c2f98944b57df1f9823
I20260925 10:30:23.993227 32202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:24.017017 32202 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:24.018401 32202 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:24.025481 32202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: Generated new TSK 0
I20260925 10:30:24.026443 32202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:24.031143 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--builtin_ntp_servers=127.12.248.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:24.396559 32216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:24.396875 32216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:24.397020 32216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:24.404664 32216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:24.404778 32216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:24.404953 32216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:24.414469 32216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:24.415937 32216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:24.417850 32216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:24.432085 32221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:24.432132 32222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:24.432158 32224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:24.432860 32216 server_base.cc:1034] running on GCE node
I20260925 10:30:24.433620 32216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:24.434891 32216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:24.436151 32216 hybrid_clock.cc:648] HybridClock initialized: now 1790332224436106 us; error 37 us; skew 500 ppm
I20260925 10:30:24.436493 32216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:24.439112 32216 webserver.cc:460] Webserver started at http://127.12.248.65:37595/ using document root <none> and password file <none>
I20260925 10:30:24.439841 32216 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:24.439977 32216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:24.440330 32216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:24.443367 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "08fe16a9aac1461591eb85bf1a0c0755"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "21f7ce626304aa1a4f05542bb0f1a430"
server_key_iv: "9b9ca8472682a16dbc3f965cc21551c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.444186 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "08fe16a9aac1461591eb85bf1a0c0755"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "21f7ce626304aa1a4f05542bb0f1a430"
server_key_iv: "9b9ca8472682a16dbc3f965cc21551c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.449401 32216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:30:24.452967 32230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.454288 32216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:24.454473 32216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "08fe16a9aac1461591eb85bf1a0c0755"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "21f7ce626304aa1a4f05542bb0f1a430"
server_key_iv: "9b9ca8472682a16dbc3f965cc21551c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.454731 32216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:24.470041 32216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:24.470854 32216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:24.471159 32216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:24.472086 32216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:24.473541 32216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:24.473663 32216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.473794 32216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:24.473897 32216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.507560 32216 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:35917
I20260925 10:30:24.507650 32342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:35917 every 8 connection(s)
I20260925 10:30:24.508939 32216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:24.511195 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32216
I20260925 10:30:24.511376 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:24.511911 13281 external_mini_cluster.cc:1428] Setting key 0bdde448492e8030652f7e019adb8e1a
I20260925 10:30:24.515067 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--builtin_ntp_servers=127.12.248.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:24.525673 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:24.511858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:59335 (local address 127.12.248.126:45745)
0925 10:30:24.517851 (+ 5993us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:24.517860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:24.517882 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:30:24.517942 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:24.517950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:24.518033 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:24.518162 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:24.519287 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.520466 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.522228 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.522668 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.522765 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:24.524573 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:24.524582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:24.524590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:24.524617 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:24.524679 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:24.524698 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:24.524789 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:24.525037 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 10:30:24.525164 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5851,"thread_start_us":5727,"threads_started":1}
I20260925 10:30:24.526700 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:24.512423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:45745 (local address 127.12.248.65:59335)
0925 10:30:24.513786 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:24.513846 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:24.517094 (+ 3248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:24.518400 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:24.518411 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:24.518506 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:24.519093 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:24.519114 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.520673 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.520677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:24.522059 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:24.522074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.522917 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.522926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:24.523274 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:24.524184 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:24.524190 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:24.524247 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:24.524356 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:24.524811 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:24.524824 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:24.524830 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:30:24.525011 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 10:30:24.525379 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":426,"threads_started":1}
I20260925 10:30:24.528196 32343 heartbeater.cc:347] Connected to a master server at 127.12.248.126:45745
I20260925 10:30:24.528533 32343 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:24.529456 32343 heartbeater.cc:511] Master 127.12.248.126:45745 requested a full tablet report, sending...
I20260925 10:30:24.531519 32157 ts_manager.cc:194] Registered new tserver with Master: 08fe16a9aac1461591eb85bf1a0c0755 (127.12.248.65:35917)
I20260925 10:30:24.532984 32157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.65:59335
W20260925 10:30:24.851711 32347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:24.852007 32347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:24.852170 32347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:24.860064 32347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:24.860175 32347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:24.860348 32347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:24.870318 32347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:24.871840 32347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:24.873746 32347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:24.888538 32352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:24.888475 32353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:24.888509 32355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:24.889477 32347 server_base.cc:1034] running on GCE node
I20260925 10:30:24.890165 32347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:24.891247 32347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:24.892539 32347 hybrid_clock.cc:648] HybridClock initialized: now 1790332224892499 us; error 45 us; skew 500 ppm
I20260925 10:30:24.892932 32347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:24.895591 32347 webserver.cc:460] Webserver started at http://127.12.248.66:45147/ using document root <none> and password file <none>
I20260925 10:30:24.896350 32347 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:24.896487 32347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:24.896826 32347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:24.899920 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "0b5b7eb135934fa5ad9341047ab93e51"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "665407e352c02e7fa19a17c0b36c00ce"
server_key_iv: "2aebcb5a81c6a74a12b53e76e8486325"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.900750 32347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "0b5b7eb135934fa5ad9341047ab93e51"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "665407e352c02e7fa19a17c0b36c00ce"
server_key_iv: "2aebcb5a81c6a74a12b53e76e8486325"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.905979 32347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:30:24.909417 32361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.911060 32347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:24.911262 32347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "0b5b7eb135934fa5ad9341047ab93e51"
format_stamp: "Formatted at 2026-09-25 10:30:24 on dist-test-slave-f9rl"
server_key: "665407e352c02e7fa19a17c0b36c00ce"
server_key_iv: "2aebcb5a81c6a74a12b53e76e8486325"
server_key_version: "encryptionkey@0"
I20260925 10:30:24.911494 32347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:24.930151 32347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:24.930980 32347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:24.931339 32347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:24.932427 32347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:24.934080 32347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:24.934207 32347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.934342 32347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:24.934432 32347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:24.971881 32347 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:35369
I20260925 10:30:24.971930 32474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:35369 every 8 connection(s)
I20260925 10:30:24.973256 32347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:24.983280 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32347
I20260925 10:30:24.983482 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:24.983928 13281 external_mini_cluster.cc:1428] Setting key 4c7e2dc978ea04558bb03dea99462ae4
I20260925 10:30:24.986778 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:24.976354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:55249 (local address 127.12.248.126:45745)
0925 10:30:24.976550 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:24.976555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:24.978009 (+ 1454us) server_negotiation.cc:407] Connection header received
0925 10:30:24.980137 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:24.980142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:24.980236 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:24.980348 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:24.981414 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.982197 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.983866 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.984304 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.984440 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:24.985957 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:24.985962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:24.985965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:24.985982 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:24.986013 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:24.986024 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:24.986116 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:24.986373 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 10:30:24.986466 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:30:24.986873 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--builtin_ntp_servers=127.12.248.84:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:24.987895 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:24.976859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:45745 (local address 127.12.248.66:55249)
0925 10:30:24.977820 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:24.977870 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:24.979857 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:24.980586 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:24.980594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:24.980683 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:24.981237 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:24.981252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.982346 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.982353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:24.983716 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:24.983728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:24.984526 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:24.984533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:24.984763 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:24.985609 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:24.985612 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:24.985669 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:24.985769 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:24.986138 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:24.986150 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:24.986158 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:30:24.986354 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 10:30:24.986749 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
I20260925 10:30:24.989331 32475 heartbeater.cc:347] Connected to a master server at 127.12.248.126:45745
I20260925 10:30:24.989688 32475 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:24.990567 32475 heartbeater.cc:511] Master 127.12.248.126:45745 requested a full tablet report, sending...
I20260925 10:30:24.992161 32157 ts_manager.cc:194] Registered new tserver with Master: 0b5b7eb135934fa5ad9341047ab93e51 (127.12.248.66:35369)
I20260925 10:30:24.993376 32157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.66:55249
W20260925 10:30:25.326860 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:25.327188 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:25.327354 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:25.334847 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:25.335006 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:25.335201 32478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:25.344570 32478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:25.346029 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:25.347993 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:25.362663 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:25.362650 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:25.363489 32483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:25.364773 32478 server_base.cc:1034] running on GCE node
I20260925 10:30:25.365516 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:25.366641 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:25.367935 32478 hybrid_clock.cc:648] HybridClock initialized: now 1790332225367875 us; error 39 us; skew 500 ppm
I20260925 10:30:25.368239 32478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:25.370548 32478 webserver.cc:460] Webserver started at http://127.12.248.67:37853/ using document root <none> and password file <none>
I20260925 10:30:25.371229 32478 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:25.371325 32478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:25.371574 32478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:25.374718 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "cdf09aeef0654a7193b481066821afe3"
format_stamp: "Formatted at 2026-09-25 10:30:25 on dist-test-slave-f9rl"
server_key: "22b445b504b84d72856c0efe04187b19"
server_key_iv: "3a6785f4750720d386771dbeeedb5e64"
server_key_version: "encryptionkey@0"
I20260925 10:30:25.375470 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "cdf09aeef0654a7193b481066821afe3"
format_stamp: "Formatted at 2026-09-25 10:30:25 on dist-test-slave-f9rl"
server_key: "22b445b504b84d72856c0efe04187b19"
server_key_iv: "3a6785f4750720d386771dbeeedb5e64"
server_key_version: "encryptionkey@0"
I20260925 10:30:25.380995 32478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:30:25.384717 32492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:25.386236 32478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:30:25.386400 32478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "cdf09aeef0654a7193b481066821afe3"
format_stamp: "Formatted at 2026-09-25 10:30:25 on dist-test-slave-f9rl"
server_key: "22b445b504b84d72856c0efe04187b19"
server_key_iv: "3a6785f4750720d386771dbeeedb5e64"
server_key_version: "encryptionkey@0"
I20260925 10:30:25.386650 32478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:25.414968 32478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:25.415901 32478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:25.416216 32478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:25.417227 32478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:25.418730 32478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:25.418816 32478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:25.418895 32478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:25.418949 32478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:25.452739 32478 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:39511
I20260925 10:30:25.452781 32606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:39511 every 8 connection(s)
I20260925 10:30:25.454113 32478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:25.464073 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32478
I20260925 10:30:25.464257 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:25.464877 13281 external_mini_cluster.cc:1428] Setting key 089e6f9f2e926758af4624d42e325133
I20260925 10:30:25.467167 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.456690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:39705 (local address 127.12.248.126:45745)
0925 10:30:25.456865 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.456869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.458298 (+ 1429us) server_negotiation.cc:407] Connection header received
0925 10:30:25.460085 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.460090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.460166 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.460258 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:25.461303 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.462243 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.463885 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.464541 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.464668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.466270 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:25.466279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:25.466283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:25.466308 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:25.466346 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:25.466363 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:25.466443 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.466723 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.466828 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:30:25.468411 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.457162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:45745 (local address 127.12.248.67:39705)
0925 10:30:25.458131 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.458167 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.459841 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.460453 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.460461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.460544 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:25.461141 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.461157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.462391 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.462395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.463734 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.463747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.464776 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.464782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.465079 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.465868 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:25.465872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:25.465937 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:25.466015 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:25.466474 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:25.466485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:25.466493 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.466709 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.467144 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":363,"threads_started":1}
I20260925 10:30:25.469911 32607 heartbeater.cc:347] Connected to a master server at 127.12.248.126:45745
I20260925 10:30:25.470186 32607 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:25.470880 32607 heartbeater.cc:511] Master 127.12.248.126:45745 requested a full tablet report, sending...
I20260925 10:30:25.472209 32157 ts_manager.cc:194] Registered new tserver with Master: cdf09aeef0654a7193b481066821afe3 (127.12.248.67:39511)
I20260925 10:30:25.472963 32157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.67:39705
I20260925 10:30:25.480706 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:25.491958 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.485299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.12.248.126:45745)
0925 10:30:25.485634 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.485643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.486033 (+ 390us) server_negotiation.cc:407] Connection header received
0925 10:30:25.486187 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.486192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.486285 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.486379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:25.486942 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.487668 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.489339 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.489815 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.489912 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.491091 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:25.491096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:25.491099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:25.491118 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:25.491151 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:25.491164 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:25.491309 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.491532 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.491604 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:30:25.496100 32157 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:25.503152 32157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:25.526736 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.516162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.12.248.66:35369)
0925 10:30:25.516747 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.516754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.517415 (+ 661us) server_negotiation.cc:407] Connection header received
0925 10:30:25.517843 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.517851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.518117 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.518314 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:25.519332 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.521345 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.524099 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525576 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.525716 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.525954 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.526175 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.526362 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":309,"threads_started":1}
I20260925 10:30:25.526925 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.515991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:35369 (local address 127.0.0.1:40850)
0925 10:30:25.517248 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.517292 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.517603 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.518614 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.518622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.518652 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:25.519104 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.519125 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.521611 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.521619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.523893 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.523907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525776 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.525781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.525991 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.526015 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.526107 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.526456 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":88,"thread_start_us":878,"threads_started":1}
I20260925 10:30:25.526963 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.516117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:39511 (local address 127.0.0.1:35642)
0925 10:30:25.518203 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.518221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.518336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.518976 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.518980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.519009 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:25.519413 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.519427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.520948 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.520953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.523489 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.523506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525671 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.525680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.525913 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.525967 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.526117 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.526460 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1648,"mutex_wait_us":163,"spinlock_wait_cycles":564992,"thread_start_us":1694,"threads_started":1}
I20260925 10:30:25.528872 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.516831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.12.248.67:39511)
0925 10:30:25.517569 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.517585 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.518461 (+ 876us) server_negotiation.cc:407] Connection header received
0925 10:30:25.518569 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.518577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.518794 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.518986 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:25.519658 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.520740 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.523729 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525015 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.525156 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.525476 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.528392 (+ 2916us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.528511 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":395,"threads_started":1}
I20260925 10:30:25.530817 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.518424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55686)
0925 10:30:25.522407 (+ 3983us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.522425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.522540 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.523113 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.523118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.523143 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:25.523586 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.523596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525577 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.525583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.527839 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.527853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.530040 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.530047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.530231 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.530263 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.530362 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.530496 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3836,"mutex_wait_us":2,"thread_start_us":3638,"threads_started":1}
I20260925 10:30:25.530990 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.518947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.12.248.65:35917)
0925 10:30:25.519650 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.519658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.522541 (+ 2883us) server_negotiation.cc:407] Connection header received
0925 10:30:25.522683 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.522691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.522913 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.523114 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:25.523793 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.525332 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.528024 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.529517 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.529660 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.529895 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.530437 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.530600 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":395,"threads_started":1}
I20260925 10:30:25.531526 32409 tablet_service.cc:1467] Processing CreateTablet for tablet 783ec4ccba8640e886302f80f146c16d (DEFAULT_TABLE table=test-table [id=3e8656d48b7a45a3b5537e830631ea21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:25.532238 32541 tablet_service.cc:1467] Processing CreateTablet for tablet 783ec4ccba8640e886302f80f146c16d (DEFAULT_TABLE table=test-table [id=3e8656d48b7a45a3b5537e830631ea21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:25.532931 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ec4ccba8640e886302f80f146c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:25.533766 32541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ec4ccba8640e886302f80f146c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:25.534547 32278 tablet_service.cc:1467] Processing CreateTablet for tablet 783ec4ccba8640e886302f80f146c16d (DEFAULT_TABLE table=test-table [id=3e8656d48b7a45a3b5537e830631ea21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:25.535888 32278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 783ec4ccba8640e886302f80f146c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:25.536787 32343 heartbeater.cc:503] Master 127.12.248.126:45745 was elected leader, sending a full tablet report...
I20260925 10:30:25.544826 32625 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Bootstrap starting.
I20260925 10:30:25.548470 32625 tablet_bootstrap.cc:654] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:25.549551 32625 log.cc:824] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:25.552076 32627 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Bootstrap starting.
I20260925 10:30:25.552579 32625 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: No bootstrap required, opened a new log
I20260925 10:30:25.552821 32625 ts_tablet_manager.cc:1397] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 10:30:25.553720 32626 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Bootstrap starting.
I20260925 10:30:25.555578 32627 tablet_bootstrap.cc:654] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:25.556967 32627 log.cc:824] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:25.557291 32626 tablet_bootstrap.cc:654] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:25.559144 32626 log.cc:824] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:25.560758 32627 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: No bootstrap required, opened a new log
I20260925 10:30:25.561057 32627 ts_tablet_manager.cc:1397] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 10:30:25.561923 32626 tablet_bootstrap.cc:492] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: No bootstrap required, opened a new log
I20260925 10:30:25.562199 32626 ts_tablet_manager.cc:1397] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260925 10:30:25.562440 32625 raft_consensus.cc:348] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.562925 32625 raft_consensus.cc:374] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:25.563026 32625 raft_consensus.cc:729] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5b7eb135934fa5ad9341047ab93e51, State: Initialized, Role: FOLLOWER
I20260925 10:30:25.563684 32625 consensus_queue.cc:260] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.564939 32475 heartbeater.cc:503] Master 127.12.248.126:45745 was elected leader, sending a full tablet report...
I20260925 10:30:25.565635 32625 ts_tablet_manager.cc:1428] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 10:30:25.571226 32627 raft_consensus.cc:348] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.571614 32627 raft_consensus.cc:374] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:25.571731 32627 raft_consensus.cc:729] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf09aeef0654a7193b481066821afe3, State: Initialized, Role: FOLLOWER
I20260925 10:30:25.572378 32627 consensus_queue.cc:260] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.573519 32607 heartbeater.cc:503] Master 127.12.248.126:45745 was elected leader, sending a full tablet report...
I20260925 10:30:25.573232 32626 raft_consensus.cc:348] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.573758 32626 raft_consensus.cc:374] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:25.573875 32626 raft_consensus.cc:729] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fe16a9aac1461591eb85bf1a0c0755, State: Initialized, Role: FOLLOWER
I20260925 10:30:25.574077 32627 ts_tablet_manager.cc:1428] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 10:30:25.574443 32626 consensus_queue.cc:260] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.576344 32626 ts_tablet_manager.cc:1428] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260925 10:30:25.587954 32632 raft_consensus.cc:482] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:25.588272 32632 raft_consensus.cc:504] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.590508 32632 leader_election.cc:288] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5b7eb135934fa5ad9341047ab93e51 (127.12.248.66:35369), 08fe16a9aac1461591eb85bf1a0c0755 (127.12.248.65:35917)
I20260925 10:30:25.597028 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.591044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:48887 (local address 127.12.248.66:35369)
0925 10:30:25.591266 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.591273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.591316 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:30:25.591397 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.591403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.591493 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.591587 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:25.592052 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.593024 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.594955 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.596060 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.596190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.596241 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.596725 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.596829 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:30:25.597148 32609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:35369 (local address 127.12.248.67:48887)
0925 10:30:25.591040 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.591071 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.591222 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.591677 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.591682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.591710 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:25.591894 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.591906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.593183 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.593189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.594781 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.594794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.596235 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.596242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.596472 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.596513 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.596634 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.596798 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 10:30:25.598198 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.590781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.12.248.67:35009)
0925 10:30:25.591488 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.591505 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:25.591609 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:25.592159 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:25.592164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:25.592189 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:25.592547 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.592558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.593814 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.593818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.595571 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:25.595587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.597421 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.597427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:25.597604 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.597628 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 10:30:25.597746 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:30:25.597875 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":357,"threads_started":1}
I20260925 10:30:25.598713 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.591483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:35009 (local address 127.12.248.65:35917)
0925 10:30:25.591735 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.591742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.591762 (+ 20us) server_negotiation.cc:407] Connection header received
0925 10:30:25.591828 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.591835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.591916 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.592014 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:25.592718 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.593639 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.595800 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.597104 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.597292 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.597380 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.597780 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.597911 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:30:25.598526 32429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ec4ccba8640e886302f80f146c16d" candidate_uuid: "cdf09aeef0654a7193b481066821afe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b7eb135934fa5ad9341047ab93e51" is_pre_election: true
I20260925 10:30:25.599211 32429 raft_consensus.cc:2454] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdf09aeef0654a7193b481066821afe3 in term 0.
I20260925 10:30:25.599923 32495 leader_election.cc:302] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b7eb135934fa5ad9341047ab93e51, cdf09aeef0654a7193b481066821afe3; no voters:
I20260925 10:30:25.599869 32298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ec4ccba8640e886302f80f146c16d" candidate_uuid: "cdf09aeef0654a7193b481066821afe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fe16a9aac1461591eb85bf1a0c0755" is_pre_election: true
I20260925 10:30:25.600371 32632 raft_consensus.cc:2781] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:25.600484 32632 raft_consensus.cc:482] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:25.600400 32298 raft_consensus.cc:2454] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdf09aeef0654a7193b481066821afe3 in term 0.
I20260925 10:30:25.600615 32632 raft_consensus.cc:3037] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:25.603066 32632 raft_consensus.cc:504] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.603762 32632 leader_election.cc:288] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5b7eb135934fa5ad9341047ab93e51 (127.12.248.66:35369), 08fe16a9aac1461591eb85bf1a0c0755 (127.12.248.65:35917)
I20260925 10:30:25.604264 32429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ec4ccba8640e886302f80f146c16d" candidate_uuid: "cdf09aeef0654a7193b481066821afe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b7eb135934fa5ad9341047ab93e51"
I20260925 10:30:25.604357 32298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "783ec4ccba8640e886302f80f146c16d" candidate_uuid: "cdf09aeef0654a7193b481066821afe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fe16a9aac1461591eb85bf1a0c0755"
I20260925 10:30:25.604485 32429 raft_consensus.cc:3037] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:25.604688 32298 raft_consensus.cc:3037] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:25.607002 32429 raft_consensus.cc:2454] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdf09aeef0654a7193b481066821afe3 in term 1.
I20260925 10:30:25.607093 32298 raft_consensus.cc:2454] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdf09aeef0654a7193b481066821afe3 in term 1.
I20260925 10:30:25.607545 32495 leader_election.cc:302] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b7eb135934fa5ad9341047ab93e51, cdf09aeef0654a7193b481066821afe3; no voters:
I20260925 10:30:25.607944 32632 raft_consensus.cc:2781] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:25.608410 32632 raft_consensus.cc:686] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 1 LEADER]: Becoming Leader. State: Replica: cdf09aeef0654a7193b481066821afe3, State: Running, Role: LEADER
I20260925 10:30:25.608959 32632 consensus_queue.cc:237] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } }
I20260925 10:30:25.613631 32157 catalog_manager.cc:5495] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdf09aeef0654a7193b481066821afe3 (127.12.248.67). New cstate: current_term: 1 leader_uuid: "cdf09aeef0654a7193b481066821afe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdf09aeef0654a7193b481066821afe3" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 } health_report { overall_health: UNKNOWN } } }
I20260925 10:30:25.631428 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.626035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.12.248.67:39511)
0925 10:30:25.626225 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.626229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.626248 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:30:25.626339 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.626342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.626422 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.626545 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:25.627074 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.627823 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.629218 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.629686 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.629780 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.630282 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:25.630798 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:25.630926 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.631084 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.631149 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:30:25.639762 32429 raft_consensus.cc:1261] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 1 FOLLOWER]: Refusing update from remote peer cdf09aeef0654a7193b481066821afe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:25.639788 32298 raft_consensus.cc:1261] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 1 FOLLOWER]: Refusing update from remote peer cdf09aeef0654a7193b481066821afe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:25.640767 32635 consensus_queue.cc:1035] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b7eb135934fa5ad9341047ab93e51" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:25.641104 32632 consensus_queue.cc:1035] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08fe16a9aac1461591eb85bf1a0c0755" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 35917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:25.670035 32156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:25.670442 32156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:25.674975 32156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95515921571f4a4ab818b047b3bb25e2: Sending DeleteTablet for 3 replicas of tablet 783ec4ccba8640e886302f80f146c16d
I20260925 10:30:25.676189 32409 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ec4ccba8640e886302f80f146c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:25 UTC) from {username='slave'} at 127.0.0.1:40850
I20260925 10:30:25.676416 32541 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ec4ccba8640e886302f80f146c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:25 UTC) from {username='slave'} at 127.0.0.1:35642
I20260925 10:30:25.677453 32278 tablet_service.cc:1514] Processing DeleteTablet for tablet 783ec4ccba8640e886302f80f146c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:25 UTC) from {username='slave'} at 127.0.0.1:55686
I20260925 10:30:25.677943 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32216
I20260925 10:30:25.689206 32649 tablet_replica.cc:330] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: stopping tablet replica
I20260925 10:30:25.690599 32649 raft_consensus.cc:2229] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:25.691684 32650 tablet_replica.cc:330] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: stopping tablet replica
I20260925 10:30:25.692238 32650 raft_consensus.cc:2229] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:25.692473 32649 raft_consensus.cc:2258] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:25.692984 32650 raft_consensus.cc:2258] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:25.695429 32649 ts_tablet_manager.cc:1905] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:25.695427 32650 ts_tablet_manager.cc:1905] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:25.700946 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.679872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.12.248.65:35917)
0925 10:30:25.680100 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.680108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.690673 (+ 10565us) server_negotiation.cc:407] Connection header received
0925 10:30:25.690907 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.690914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.691072 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.691197 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:25.692154 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.694004 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.696259 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.697046 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.697171 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.698886 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:25.698899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:25.698917 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:25.699788 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:25.699866 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:25.699893 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:25.699998 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.700347 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.700487 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:30:25.702533 32338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:25.702976 32649 ts_tablet_manager.cc:1918] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:25.703169 32648 tablet_replica.cc:330] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: stopping tablet replica
I20260925 10:30:25.703214 32649 log.cc:1196] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/783ec4ccba8640e886302f80f146c16d
I20260925 10:30:25.703840 32648 raft_consensus.cc:2229] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:25.704039 32649 ts_tablet_manager.cc:1939] T 783ec4ccba8640e886302f80f146c16d P cdf09aeef0654a7193b481066821afe3: Deleting consensus metadata
I20260925 10:30:25.704594 32648 raft_consensus.cc:2258] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:25.708173 32144 catalog_manager.cc:4841] TS cdf09aeef0654a7193b481066821afe3 (127.12.248.67:39511): tablet 783ec4ccba8640e886302f80f146c16d (table test-table [id=3e8656d48b7a45a3b5537e830631ea21]) successfully deleted
I20260925 10:30:25.708528 32648 ts_tablet_manager.cc:1905] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:25.716946 32648 ts_tablet_manager.cc:1918] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:25.717166 32648 log.cc:1196] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/783ec4ccba8640e886302f80f146c16d
I20260925 10:30:25.717850 32648 ts_tablet_manager.cc:1939] T 783ec4ccba8640e886302f80f146c16d P 0b5b7eb135934fa5ad9341047ab93e51: Deleting consensus metadata
I20260925 10:30:25.720321 32143 catalog_manager.cc:4841] TS 0b5b7eb135934fa5ad9341047ab93e51 (127.12.248.66:35369): tablet 783ec4ccba8640e886302f80f146c16d (table test-table [id=3e8656d48b7a45a3b5537e830631ea21]) successfully deleted
I20260925 10:30:25.799360 32650 ts_tablet_manager.cc:1918] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:25.799737 32650 log.cc:1196] T 783ec4ccba8640e886302f80f146c16d P 08fe16a9aac1461591eb85bf1a0c0755: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/783ec4ccba8640e886302f80f146c16d
I20260925 10:30:25.799998 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32216
W20260925 10:30:25.818184 32144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.248.65:35917 (error 108)
I20260925 10:30:25.818642 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32347
I20260925 10:30:25.820429 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.819908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55700)
0925 10:30:25.820116 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.820252 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 10:30:25.820706 32144 catalog_manager.cc:4584] TS 08fe16a9aac1461591eb85bf1a0c0755 (127.12.248.65:35917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 783ec4ccba8640e886302f80f146c16d: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
I20260925 10:30:25.827051 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.819603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.12.248.66:35369)
0925 10:30:25.819831 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.819835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.819851 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:30:25.819904 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.819908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.820039 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.820144 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:25.820820 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.822067 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.823562 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.824020 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.824176 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.825380 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:25.825389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:25.825405 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:25.826204 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:25.826268 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:25.826296 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:25.826429 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.826562 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.826679 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:30:25.827818 32469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:25.874943 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.874346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55708)
0925 10:30:25.874595 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.874723 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 10:30:25.876176 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.875775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55710)
0925 10:30:25.875942 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.876029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:30:25.936877 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.936239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55712)
0925 10:30:25.936550 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.936674 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260925 10:30:25.938195 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.937746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55720)
0925 10:30:25.937971 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:25.938055 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 10:30:25.943033 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32347
I20260925 10:30:25.961699 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32478
I20260925 10:30:25.970710 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:25.962811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.12.248.67:39511)
0925 10:30:25.963031 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:25.963037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:25.963061 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:30:25.963144 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:25.963149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:25.963252 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:25.963384 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:25.963931 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.965143 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.966891 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:25.967600 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:25.967746 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:25.968836 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:25.968848 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:25.968870 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:25.969840 (+ 970us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:25.969904 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:25.969930 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:25.970044 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:25.970188 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 10:30:25.970300 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 10:30:25.971469 32602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:26.019055 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:26.018452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55736)
0925 10:30:26.018709 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:26.018826 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:30:26.020210 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:26.019767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55752)
0925 10:30:26.019993 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:26.020077 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 10:30:26.097582 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32478
I20260925 10:30:26.117529 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32126
I20260925 10:30:26.118759 32187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:26.231565 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:26.231010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:35917 (local address 127.0.0.1:55762)
0925 10:30:26.231256 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:26.231352 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.65:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:30:26.231526 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32126
2026-09-25T10:30:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2846 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:26 dist-test-slave-f9rl krb5kdc[32659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:26 dist-test-slave-f9rl krb5kdc[32659](info): set up 2 sockets
Sep 25 10:30:26 dist-test-slave-f9rl krb5kdc[32659](info): commencing operation
krb5kdc: starting...
W20260925 10:30:27.351300 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:27 dist-test-slave-f9rl krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:30:27.621258 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:42421
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:27.982788 32672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:27.983170 32672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:27.983333 32672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:27.991830 32672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:27.992028 32672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:27.992151 32672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:27.992266 32672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:27.992362 32672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:28.004132 32672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:42421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:28.005750 32672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:28.007220 32672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:28.020661 32678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:28.020798 32677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:28.020754 32680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:28.021629 32672 server_base.cc:1034] running on GCE node
I20260925 10:30:28.022810 32672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:28.025003 32672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:28.026345 32672 hybrid_clock.cc:648] HybridClock initialized: now 1790332228026276 us; error 60 us; skew 500 ppm
I20260925 10:30:28.026859 32672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:28 dist-test-slave-f9rl krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:28.030426 32672 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:30:28.032132 32672 webserver.cc:460] Webserver started at http://127.12.248.126:39199/ using document root <none> and password file <none>
I20260925 10:30:28.032892 32672 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:28.033023 32672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:28.033365 32672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:28.037043 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "33a2cd6f6ee545cc8424062ceba131f6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4b04b233f71d666acd96fcc985e80a1a"
server_key_iv: "1a82f8ea24ee0bf618b9bec1a161f099"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.037993 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "33a2cd6f6ee545cc8424062ceba131f6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4b04b233f71d666acd96fcc985e80a1a"
server_key_iv: "1a82f8ea24ee0bf618b9bec1a161f099"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.043891 32672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:28.048058 32687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:28.049875 32672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:30:28.050099 32672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "33a2cd6f6ee545cc8424062ceba131f6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4b04b233f71d666acd96fcc985e80a1a"
server_key_iv: "1a82f8ea24ee0bf618b9bec1a161f099"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.050331 32672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:28.083319 32672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:28.087961 32672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:28.088299 32672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:28.108685 32672 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:42421
I20260925 10:30:28.108860 32739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:42421 every 8 connection(s)
I20260925 10:30:28.110452 32672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:28.114833 32740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:28.117569 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32672
I20260925 10:30:28.117880 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:28.118395 13281 external_mini_cluster.cc:1428] Setting key 612e9819dd374c40e7bcd6e3afc22030
I20260925 10:30:28.128782 32740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: Bootstrap starting.
I20260925 10:30:28.132758 32740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:28.134301 32740 log.cc:824] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:30:28 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:28.137940 32740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: No bootstrap required, opened a new log
I20260925 10:30:28.145511 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:28.120839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.12.248.126:42421)
0925 10:30:28.121761 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:28.121779 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:28.121833 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:30:28.124361 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:28.124396 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:28.125026 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:28.125556 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:28.127373 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.129397 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.130884 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.131989 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.132184 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:28.137184 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:28.137192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:28.137210 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:28.137252 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:28.140725 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:28.141613 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:28.141620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:28.141627 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:28.141841 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:28.142308 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:28.142312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:28.142315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:28.142942 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:28.143292 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:28.143808 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 10:30:28.144290 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":377,"threads_started":1}
I20260925 10:30:28.147567 32740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } }
I20260925 10:30:28.148090 32740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:28.148244 32740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a2cd6f6ee545cc8424062ceba131f6, State: Initialized, Role: FOLLOWER
I20260925 10:30:28.148976 32740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } }
I20260925 10:30:28.149326 32740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:28.149510 32740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:28.149706 32740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:28.152139 32740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } }
I20260925 10:30:28.152745 32740 leader_election.cc:302] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33a2cd6f6ee545cc8424062ceba131f6; no voters:
I20260925 10:30:28.153338 32740 leader_election.cc:288] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:28.153594 32745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:28.154896 32745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [term 1 LEADER]: Becoming Leader. State: Replica: 33a2cd6f6ee545cc8424062ceba131f6, State: Running, Role: LEADER
I20260925 10:30:28.155570 32745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } }
I20260925 10:30:28.156039 32740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:28.158655 32746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33a2cd6f6ee545cc8424062ceba131f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } } }
I20260925 10:30:28.159103 32746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:28.159585 32747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33a2cd6f6ee545cc8424062ceba131f6. Latest consensus state: current_term: 1 leader_uuid: "33a2cd6f6ee545cc8424062ceba131f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33a2cd6f6ee545cc8424062ceba131f6" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 42421 } } }
I20260925 10:30:28.159885 32747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:28.161597 32751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:28.168383 32751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:28.178573 32751 catalog_manager.cc:1347] Generated new cluster ID: e38a4016cd364ea890740fdf86181415
I20260925 10:30:28.178722 32751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:28.190032 32751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:28.191478 32751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:28.197475 32751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: Generated new TSK 0
I20260925 10:30:28.198625 32751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.248.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
Entry for principal kudu/127.12.248.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab.
I20260925 10:30:28.273664 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:28.614450 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:28.614851 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:28.615059 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:28.622416 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:28.622578 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:28.622778 32766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:28.632606 32766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.65.keytab
--principal=kudu/127.12.248.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:28.634053 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:28.635474 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:28.649708 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:28.649806 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:28.649802 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:28.650758 32766 server_base.cc:1034] running on GCE node
I20260925 10:30:28.651387 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:28.652407 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:28.653635 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790332228653563 us; error 37 us; skew 500 ppm
I20260925 10:30:28.654047 32766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:28 dist-test-slave-f9rl krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:28.657930 32766 init.cc:377] Logged in from keytab as kudu/127.12.248.65@KRBTEST.COM (short username kudu)
I20260925 10:30:28.659652 32766 webserver.cc:460] Webserver started at http://127.12.248.65:38567/ using document root <none> and password file <none>
I20260925 10:30:28.660462 32766 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:28.660621 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:28.661083 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:28.665083 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "479b0e4a94d54dbcaace5812ba447cc6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4e6040cab43b7b8a4037a372f4c74da1"
server_key_iv: "60a3059893f61c3e110accc8682a4b38"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.666015 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "479b0e4a94d54dbcaace5812ba447cc6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4e6040cab43b7b8a4037a372f4c74da1"
server_key_iv: "60a3059893f61c3e110accc8682a4b38"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.671738 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:28.675953 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:28.677848 32766 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 10:30:28.678042 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "479b0e4a94d54dbcaace5812ba447cc6"
format_stamp: "Formatted at 2026-09-25 10:30:28 on dist-test-slave-f9rl"
server_key: "4e6040cab43b7b8a4037a372f4c74da1"
server_key_iv: "60a3059893f61c3e110accc8682a4b38"
server_key_version: "encryptionkey@0"
I20260925 10:30:28.678279 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:28.696488 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:28.701915 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:28.702277 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:28.703372 32766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:28.704838 32766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:28.704911 32766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:28.705039 32766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:28.705108 32766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:28.740638 32766 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:40881
I20260925 10:30:28.740685 426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:40881 every 8 connection(s)
I20260925 10:30:28.742238 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:28.752460 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 32766
I20260925 10:30:28.752656 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:28.753135 13281 external_mini_cluster.cc:1428] Setting key 644a6ae09e1151a06a1d8958deed678b
Sep 25 10:30:28 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332228, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.65@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:28.765586 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:28.745520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:58451 (local address 127.12.248.126:42421)
0925 10:30:28.746024 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:28.746044 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:28.747120 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 10:30:28.749064 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:28.749069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:28.749259 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:28.749375 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:28.751102 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.751881 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.753917 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.754571 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.754680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:28.758915 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:28.758920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:28.758924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:28.758961 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:28.761904 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:28.762899 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:28.762905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:28.762910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:28.763049 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:28.763592 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:28.763596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:28.763598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:28.763911 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:28.764081 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:28.764897 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 10:30:28.765157 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260925 10:30:28.766666 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:28.746083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42421 (local address 127.12.248.65:58451)
0925 10:30:28.746933 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:28.746975 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:28.748743 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:28.749709 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:28.749718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:28.750278 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:28.750899 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:28.750919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.752064 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.752068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:28.753664 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:28.753741 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:28.754838 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:28.754846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:28.755231 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:28.756097 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:28.756127 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:28.758620 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:28.762094 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:28.762104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:28.762135 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:28.762747 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:28.763178 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:28.763183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:28.763187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:28.763457 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:28.764054 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:28.764063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:28.764465 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 10:30:28.764786 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 10:30:28.765397 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":382,"threads_started":1}
I20260925 10:30:28.768467 427 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42421
I20260925 10:30:28.768831 427 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:28.769728 427 heartbeater.cc:511] Master 127.12.248.126:42421 requested a full tablet report, sending...
I20260925 10:30:28.772428 32704 ts_manager.cc:194] Registered new tserver with Master: 479b0e4a94d54dbcaace5812ba447cc6 (127.12.248.65:40881)
I20260925 10:30:28.774111 32704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.65@KRBTEST.COM'} at 127.12.248.65:58451
WARNING: no policy specified for kudu/127.12.248.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
Entry for principal kudu/127.12.248.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab.
I20260925 10:30:28.814451 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:29.164789 433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:29.165117 433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:29.165216 433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:29.172708 433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:29.172832 433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:29.172950 433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:29.183588 433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.66.keytab
--principal=kudu/127.12.248.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:29.184968 433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:29.186363 433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:29.199875 438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:29.199846 441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:29.200047 439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:29.201071 433 server_base.cc:1034] running on GCE node
I20260925 10:30:29.201874 433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:29.202956 433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:29.204219 433 hybrid_clock.cc:648] HybridClock initialized: now 1790332229204133 us; error 74 us; skew 500 ppm
I20260925 10:30:29.204658 433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:29 dist-test-slave-f9rl krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:29.209565 433 init.cc:377] Logged in from keytab as kudu/127.12.248.66@KRBTEST.COM (short username kudu)
I20260925 10:30:29.211213 433 webserver.cc:460] Webserver started at http://127.12.248.66:42333/ using document root <none> and password file <none>
I20260925 10:30:29.211989 433 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:29.212128 433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:29.212474 433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:29.215624 433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "f6c5c4442c334f4eb5a2adf4bddb0998"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "7c538dd808a9b300b364c3fde62a37b4"
server_key_iv: "6ca1c15c87433f5c21ceaa1757085d91"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.216475 433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "f6c5c4442c334f4eb5a2adf4bddb0998"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "7c538dd808a9b300b364c3fde62a37b4"
server_key_iv: "6ca1c15c87433f5c21ceaa1757085d91"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.222194 433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:30:29.226094 448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.227612 433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:30:29.227799 433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "f6c5c4442c334f4eb5a2adf4bddb0998"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "7c538dd808a9b300b364c3fde62a37b4"
server_key_iv: "6ca1c15c87433f5c21ceaa1757085d91"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.227983 433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:29.248589 433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:29.253110 433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:29.253384 433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:29.254630 433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:29.256201 433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:29.256281 433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.256438 433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:29.256496 433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.291486 433 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:36065
I20260925 10:30:29.291584 560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:36065 every 8 connection(s)
I20260925 10:30:29.292868 433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:29.301265 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 433
I20260925 10:30:29.301522 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:29.302166 13281 external_mini_cluster.cc:1428] Setting key 5679a7f22283992a994ee9d7cc001d9e
Sep 25 10:30:29 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.66@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:29.315032 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.295577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:46023 (local address 127.12.248.126:42421)
0925 10:30:29.296037 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.296042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.297024 (+ 982us) server_negotiation.cc:407] Connection header received
0925 10:30:29.298930 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.298935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.299005 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.299106 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:29.300547 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.301291 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.303551 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.304066 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.304209 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.308118 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:29.308123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:29.308126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:29.308159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:29.311231 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.312158 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.312164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.312166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.312311 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.312854 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.312859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.312861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.313204 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:29.313380 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.314258 (+ 878us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.314519 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260925 10:30:29.316180 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.296141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42421 (local address 127.12.248.66:46023)
0925 10:30:29.296851 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:29.296889 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:29.298665 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:29.299327 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:29.299335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:29.299829 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:29.300393 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.300409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.301514 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.301522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.303396 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.303408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.304312 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.304318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.304548 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.305299 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:29.305327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:29.307848 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:29.311433 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:29.311444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:29.311474 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:29.311992 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:29.312442 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:29.312447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:29.312451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:29.312692 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:29.313358 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:29.313367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:29.313841 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 10:30:29.314151 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 10:30:29.314803 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":270,"threads_started":1}
I20260925 10:30:29.317994 561 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42421
I20260925 10:30:29.318348 561 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:29.319157 561 heartbeater.cc:511] Master 127.12.248.126:42421 requested a full tablet report, sending...
I20260925 10:30:29.320783 32704 ts_manager.cc:194] Registered new tserver with Master: f6c5c4442c334f4eb5a2adf4bddb0998 (127.12.248.66:36065)
I20260925 10:30:29.321743 32704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.66@KRBTEST.COM'} at 127.12.248.66:46023
WARNING: no policy specified for kudu/127.12.248.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
Entry for principal kudu/127.12.248.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab.
I20260925 10:30:29.361665 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:29.706137 567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:29.706489 567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:29.706650 567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:29.714120 567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:29.714262 567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:29.714484 567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:29.724866 567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.67.keytab
--principal=kudu/127.12.248.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:42421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:29.726333 567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:29.727751 567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:29.742498 574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:29.742654 576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:29.743129 572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:29.744270 567 server_base.cc:1034] running on GCE node
I20260925 10:30:29.745018 567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:29.746253 567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:29.747547 567 hybrid_clock.cc:648] HybridClock initialized: now 1790332229747484 us; error 57 us; skew 500 ppm
I20260925 10:30:29.747927 567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:29 dist-test-slave-f9rl krb5kdc[32659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:29.751570 567 init.cc:377] Logged in from keytab as kudu/127.12.248.67@KRBTEST.COM (short username kudu)
I20260925 10:30:29.753537 567 webserver.cc:460] Webserver started at http://127.12.248.67:36847/ using document root <none> and password file <none>
I20260925 10:30:29.754468 567 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:29.754627 567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:29.754993 567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:29.758404 567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "5f93d834f6564727bb36524a7adacb07"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "c208a5524a4532e3b03a4d41c2f5bbfc"
server_key_iv: "39ebc232b58037b9124193adf5232f0a"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.759172 567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "5f93d834f6564727bb36524a7adacb07"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "c208a5524a4532e3b03a4d41c2f5bbfc"
server_key_iv: "39ebc232b58037b9124193adf5232f0a"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.765074 567 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 10:30:29.769202 583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.770893 567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 10:30:29.771116 567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "5f93d834f6564727bb36524a7adacb07"
format_stamp: "Formatted at 2026-09-25 10:30:29 on dist-test-slave-f9rl"
server_key: "c208a5524a4532e3b03a4d41c2f5bbfc"
server_key_iv: "39ebc232b58037b9124193adf5232f0a"
server_key_version: "encryptionkey@0"
I20260925 10:30:29.771346 567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:29.777603 427 heartbeater.cc:503] Master 127.12.248.126:42421 was elected leader, sending a full tablet report...
I20260925 10:30:29.788089 567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:29.792896 567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:29.793252 567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:29.794369 567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:29.795938 567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:29.796020 567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.796213 567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:29.796268 567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:29.832046 567 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:38309
I20260925 10:30:29.832083 695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:38309 every 8 connection(s)
I20260925 10:30:29.833556 567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:29.838950 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 567
I20260925 10:30:29.839164 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:29.839613 13281 external_mini_cluster.cc:1428] Setting key e8228f78606f18c99a10676be8df91d6
Sep 25 10:30:29 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332229, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.67@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:29.855674 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.836427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:59519 (local address 127.12.248.126:42421)
0925 10:30:29.836956 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.836961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.838199 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 10:30:29.840384 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.840392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.840481 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.840600 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:29.842558 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.843362 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.845006 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.845524 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.845628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.849241 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:29.849246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:29.849250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:29.849284 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:29.852095 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.852875 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.852881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.852885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.853016 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.853554 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.853558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.853560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.853991 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:29.854151 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.855134 (+ 983us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.855384 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
I20260925 10:30:29.856755 699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.837033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:42421 (local address 127.12.248.67:59519)
0925 10:30:29.838014 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:29.838057 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:29.839979 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:29.841044 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:29.841054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:29.841732 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:29.842383 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.842414 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.843506 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.843510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.844857 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.844870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.845808 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.845816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.846064 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.846741 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:30:29.846763 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:29.848998 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:29.852247 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:29.852254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:29.852269 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:29.852737 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:29.853156 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:30:29.853160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:30:29.853163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:30:29.853416 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:30:29.854143 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:29.854153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:29.854648 (+ 495us) client_negotiation.cc:769] Sending connection context
0925 10:30:29.854928 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 10:30:29.855532 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":490,"threads_started":1}
I20260925 10:30:29.858278 696 heartbeater.cc:347] Connected to a master server at 127.12.248.126:42421
I20260925 10:30:29.858598 696 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:29.859380 696 heartbeater.cc:511] Master 127.12.248.126:42421 requested a full tablet report, sending...
I20260925 10:30:29.860942 32704 ts_manager.cc:194] Registered new tserver with Master: 5f93d834f6564727bb36524a7adacb07 (127.12.248.67:38309)
I20260925 10:30:29.861848 32704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.248.67@KRBTEST.COM'} at 127.12.248.67:59519
I20260925 10:30:29.868110 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:29.887331 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.872648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.12.248.126:42421)
0925 10:30:29.872933 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.872938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.873411 (+ 473us) server_negotiation.cc:407] Connection header received
0925 10:30:29.873629 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.873633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.873704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.873807 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:29.875312 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.876266 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.877719 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.878141 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.878251 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.881022 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:29.881027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:29.881030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:29.881071 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:29.884160 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.885032 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.885036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.885039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.885176 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:29.885646 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:29.885649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:29.885651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:29.886008 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:29.886219 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.886714 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.886938 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260925 10:30:29.892057 32704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:29.899380 32704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:29.948722 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.913259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.12.248.65:40881)
0925 10:30:29.939135 (+ 25876us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.939148 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.939191 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:30:29.939333 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.939346 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.940028 (+ 682us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.940242 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:29.941044 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.942746 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.945697 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.947196 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.947335 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.947596 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.948174 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.948355 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25623,"thread_start_us":26391,"threads_started":1}
I20260925 10:30:29.949280 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.912896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:40881 (local address 127.0.0.1:58148)
0925 10:30:29.938500 (+ 25604us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:29.938553 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:29.938805 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:29.940289 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:29.940297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:29.940331 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:29.940838 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.940858 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.942858 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.942866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.945520 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.945536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.947755 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.947762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.947917 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.947953 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:30:29.948134 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 10:30:29.948347 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25293,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":28379,"threads_started":1}
I20260925 10:30:29.949563 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.914240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.12.248.67:38309)
0925 10:30:29.939432 (+ 25192us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.939444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.940322 (+ 878us) server_negotiation.cc:407] Connection header received
0925 10:30:29.940629 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.940636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.940880 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.941034 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:29.942063 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.944017 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.946213 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.947638 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.947787 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.947998 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.949041 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.949208 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24971,"thread_start_us":25222,"threads_started":1}
I20260925 10:30:29.949671 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.913663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:36065 (local address 127.0.0.1:49694)
0925 10:30:29.940848 (+ 27185us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:29.940862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:29.941007 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:29.941674 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:29.941683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:29.941713 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:29.942215 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.942227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.944157 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.944163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.946325 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.946339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.948707 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.948714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.948906 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.948936 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 10:30:29.949030 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:30:29.949151 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27085,"mutex_wait_us":118,"thread_start_us":24361,"threads_started":1}
I20260925 10:30:29.949672 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.914488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.12.248.66:36065)
0925 10:30:29.938197 (+ 23709us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:29.938210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:29.940977 (+ 2767us) server_negotiation.cc:407] Connection header received
0925 10:30:29.941180 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:29.941188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:29.941420 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:29.941663 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:29.942385 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.944002 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.946752 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.948228 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.948357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.948544 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:29.949095 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 10:30:29.949297 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23430,"thread_start_us":23725,"threads_started":1}
I20260925 10:30:29.950268 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:29.913231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38309 (local address 127.0.0.1:44620)
0925 10:30:29.940205 (+ 26974us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:29.940224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:29.940423 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:29.941210 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:29.941214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:29.941236 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:29.941841 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.941857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.944196 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.944200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.946036 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:29.946052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:29.948146 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:29.948152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:29.948327 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:29.948360 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:30:29.948430 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:30:29.948570 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26812,"mutex_wait_us":7,"thread_start_us":26605,"threads_started":1}
I20260925 10:30:29.953490 361 tablet_service.cc:1467] Processing CreateTablet for tablet 30faee43076e4fbbb3afc94183079cda (DEFAULT_TABLE table=test-table [id=7c988671e6c340988528803a01c4498e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:29.953408 496 tablet_service.cc:1467] Processing CreateTablet for tablet 30faee43076e4fbbb3afc94183079cda (DEFAULT_TABLE table=test-table [id=7c988671e6c340988528803a01c4498e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:29.954793 496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30faee43076e4fbbb3afc94183079cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:29.954875 361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30faee43076e4fbbb3afc94183079cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:29.961696 631 tablet_service.cc:1467] Processing CreateTablet for tablet 30faee43076e4fbbb3afc94183079cda (DEFAULT_TABLE table=test-table [id=7c988671e6c340988528803a01c4498e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:29.963526 631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30faee43076e4fbbb3afc94183079cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:29.967410 716 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Bootstrap starting.
I20260925 10:30:29.970214 717 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Bootstrap starting.
I20260925 10:30:29.973407 716 tablet_bootstrap.cc:654] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:29.974602 717 tablet_bootstrap.cc:654] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:29.975080 716 log.cc:824] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:29.976058 717 log.cc:824] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:29.979487 716 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: No bootstrap required, opened a new log
I20260925 10:30:29.978926 720 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Bootstrap starting.
I20260925 10:30:29.979888 716 ts_tablet_manager.cc:1397] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260925 10:30:29.983152 720 tablet_bootstrap.cc:654] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:29.984246 720 log.cc:824] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:29.987572 720 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: No bootstrap required, opened a new log
I20260925 10:30:29.987901 720 ts_tablet_manager.cc:1397] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 10:30:29.988476 717 tablet_bootstrap.cc:492] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: No bootstrap required, opened a new log
I20260925 10:30:29.988777 717 ts_tablet_manager.cc:1397] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.015s
I20260925 10:30:29.994338 716 raft_consensus.cc:348] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:29.994952 716 raft_consensus.cc:374] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:29.995087 716 raft_consensus.cc:729] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c5c4442c334f4eb5a2adf4bddb0998, State: Initialized, Role: FOLLOWER
I20260925 10:30:29.995406 720 raft_consensus.cc:348] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:29.995855 720 raft_consensus.cc:374] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:29.995949 720 raft_consensus.cc:729] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f93d834f6564727bb36524a7adacb07, State: Initialized, Role: FOLLOWER
I20260925 10:30:29.995965 716 consensus_queue.cc:260] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:29.996594 720 consensus_queue.cc:260] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:29.998561 696 heartbeater.cc:503] Master 127.12.248.126:42421 was elected leader, sending a full tablet report...
I20260925 10:30:29.999111 720 ts_tablet_manager.cc:1428] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260925 10:30:30.000130 561 heartbeater.cc:503] Master 127.12.248.126:42421 was elected leader, sending a full tablet report...
I20260925 10:30:30.000815 716 ts_tablet_manager.cc:1428] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260925 10:30:30.001201 717 raft_consensus.cc:348] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:30.001700 717 raft_consensus.cc:374] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:30.001823 717 raft_consensus.cc:729] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479b0e4a94d54dbcaace5812ba447cc6, State: Initialized, Role: FOLLOWER
I20260925 10:30:30.002478 717 consensus_queue.cc:260] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:30.004282 717 ts_tablet_manager.cc:1428] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260925 10:30:30.045083 562 tablet.cc:2369] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:30:30.085309 697 tablet.cc:2369] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:30.195737 722 raft_consensus.cc:482] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:30.196125 722 raft_consensus.cc:504] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:30.198079 722 leader_election.cc:288] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479b0e4a94d54dbcaace5812ba447cc6 (127.12.248.65:40881), 5f93d834f6564727bb36524a7adacb07 (127.12.248.67:38309)
I20260925 10:30:30.205004 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.198764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:58313 (local address 127.12.248.67:38309)
0925 10:30:30.198984 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.198992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.199433 (+ 441us) server_negotiation.cc:407] Connection header received
0925 10:30:30.199678 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.199682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.199743 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.199846 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:30.200451 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.201191 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.203196 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.204008 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.204207 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.204295 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.204600 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.204739 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:30:30.205092 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.198432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:49781 (local address 127.12.248.65:40881)
0925 10:30:30.198666 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.198673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.199578 (+ 905us) server_negotiation.cc:407] Connection header received
0925 10:30:30.199791 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.199795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.199854 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.199956 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:30.200576 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.201290 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.203237 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.204078 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.204321 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.204402 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.204694 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.204835 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:30:30.205180 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.198445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:38309 (local address 127.12.248.66:58313)
0925 10:30:30.199314 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:30.199332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:30.199509 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:30.199864 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:30.199869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:30.199893 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:30.200296 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:30.200308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.201329 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.201336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:30.203013 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:30.203027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.204263 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.204269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:30.204424 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.204453 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 10:30:30.204570 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:30:30.204717 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":6,"thread_start_us":553,"threads_started":1}
I20260925 10:30:30.205665 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.198492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:40881 (local address 127.12.248.66:49781)
0925 10:30:30.199460 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:30.199479 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:30.199642 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:30.199974 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:30.199979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:30.200017 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:30.200426 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:30.200435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.201464 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.201469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:30.203039 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:30.203052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.204364 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.204371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:30.204536 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.204559 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:30:30.204666 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:30:30.204806 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":27,"thread_start_us":597,"threads_started":1}
I20260925 10:30:30.206351 651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30faee43076e4fbbb3afc94183079cda" candidate_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f93d834f6564727bb36524a7adacb07" is_pre_election: true
I20260925 10:30:30.206714 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30faee43076e4fbbb3afc94183079cda" candidate_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479b0e4a94d54dbcaace5812ba447cc6" is_pre_election: true
I20260925 10:30:30.206982 651 raft_consensus.cc:2454] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c5c4442c334f4eb5a2adf4bddb0998 in term 0.
I20260925 10:30:30.207301 381 raft_consensus.cc:2454] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c5c4442c334f4eb5a2adf4bddb0998 in term 0.
I20260925 10:30:30.207787 452 leader_election.cc:302] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f93d834f6564727bb36524a7adacb07, f6c5c4442c334f4eb5a2adf4bddb0998; no voters:
I20260925 10:30:30.208242 722 raft_consensus.cc:2781] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:30.208427 722 raft_consensus.cc:482] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:30.208556 722 raft_consensus.cc:3037] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:30.210872 722 raft_consensus.cc:504] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:30.211534 722 leader_election.cc:288] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [CANDIDATE]: Term 1 election: Requested vote from peers 479b0e4a94d54dbcaace5812ba447cc6 (127.12.248.65:40881), 5f93d834f6564727bb36524a7adacb07 (127.12.248.67:38309)
I20260925 10:30:30.211956 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30faee43076e4fbbb3afc94183079cda" candidate_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479b0e4a94d54dbcaace5812ba447cc6"
I20260925 10:30:30.212160 381 raft_consensus.cc:3037] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:30.212162 651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30faee43076e4fbbb3afc94183079cda" candidate_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f93d834f6564727bb36524a7adacb07"
I20260925 10:30:30.212399 651 raft_consensus.cc:3037] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:30.214612 381 raft_consensus.cc:2454] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c5c4442c334f4eb5a2adf4bddb0998 in term 1.
I20260925 10:30:30.214757 651 raft_consensus.cc:2454] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c5c4442c334f4eb5a2adf4bddb0998 in term 1.
I20260925 10:30:30.215095 450 leader_election.cc:302] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479b0e4a94d54dbcaace5812ba447cc6, f6c5c4442c334f4eb5a2adf4bddb0998; no voters:
I20260925 10:30:30.215447 722 raft_consensus.cc:2781] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:30.215935 722 raft_consensus.cc:686] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 1 LEADER]: Becoming Leader. State: Replica: f6c5c4442c334f4eb5a2adf4bddb0998, State: Running, Role: LEADER
I20260925 10:30:30.216470 722 consensus_queue.cc:237] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } }
I20260925 10:30:30.221534 32702 catalog_manager.cc:5495] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c5c4442c334f4eb5a2adf4bddb0998 (127.12.248.66). New cstate: current_term: 1 leader_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c5c4442c334f4eb5a2adf4bddb0998" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36065 } health_report { overall_health: HEALTHY } } }
W20260925 10:30:30.245287 428 tablet.cc:2369] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:30:30.255416 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.249797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.12.248.66:36065)
0925 10:30:30.250041 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.250049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.250113 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:30:30.250301 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.250305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.250395 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.250518 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:30.250873 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.251601 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.253079 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.253618 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.253719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.254223 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:30.254822 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:30.254974 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.255032 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.255102 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:30:30.264262 381 raft_consensus.cc:1261] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 1 FOLLOWER]: Refusing update from remote peer f6c5c4442c334f4eb5a2adf4bddb0998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:30.264271 651 raft_consensus.cc:1261] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 1 FOLLOWER]: Refusing update from remote peer f6c5c4442c334f4eb5a2adf4bddb0998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:30.265427 727 consensus_queue.cc:1035] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f93d834f6564727bb36524a7adacb07" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:30.266155 722 consensus_queue.cc:1035] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "479b0e4a94d54dbcaace5812ba447cc6" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:30.279592 730 mvcc.cc:205] Tried to move back new op lower bound from 7333200815151308800 to 7333200814971011072. Current Snapshot: MvccSnapshot[applied={T|T < 7333200815151308800}]
I20260925 10:30:30.295351 32704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:30.295650 32704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:30.299424 32704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33a2cd6f6ee545cc8424062ceba131f6: Sending DeleteTablet for 3 replicas of tablet 30faee43076e4fbbb3afc94183079cda
I20260925 10:30:30.300648 361 tablet_service.cc:1514] Processing DeleteTablet for tablet 30faee43076e4fbbb3afc94183079cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:30 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:58148
I20260925 10:30:30.300648 631 tablet_service.cc:1514] Processing DeleteTablet for tablet 30faee43076e4fbbb3afc94183079cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:30 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:44620
I20260925 10:30:30.303982 496 tablet_service.cc:1514] Processing DeleteTablet for tablet 30faee43076e4fbbb3afc94183079cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:30 UTC) from {username='kudu', principal='kudu/127.12.248.126@KRBTEST.COM'} at 127.0.0.1:49694
I20260925 10:30:30.306121 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32766
I20260925 10:30:30.310011 742 tablet_replica.cc:330] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: stopping tablet replica
I20260925 10:30:30.310531 742 raft_consensus.cc:2229] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:30.311308 742 raft_consensus.cc:2258] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:30.315285 741 tablet_replica.cc:330] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: stopping tablet replica
I20260925 10:30:30.315836 741 raft_consensus.cc:2229] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:30.316512 741 raft_consensus.cc:2258] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:30.317571 740 tablet_replica.cc:330] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: stopping tablet replica
I20260925 10:30:30.318109 740 raft_consensus.cc:2229] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:30.318728 740 raft_consensus.cc:2258] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:30.321695 740 ts_tablet_manager.cc:1905] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:30.322007 742 ts_tablet_manager.cc:1905] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:30.329208 740 ts_tablet_manager.cc:1918] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:30.331673 741 ts_tablet_manager.cc:1905] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:30.333949 742 ts_tablet_manager.cc:1918] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 25 10:30:30 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.65@KRBTEST.COM
I20260925 10:30:30.334201 742 log.cc:1196] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/30faee43076e4fbbb3afc94183079cda
I20260925 10:30:30.334812 742 ts_tablet_manager.cc:1939] T 30faee43076e4fbbb3afc94183079cda P f6c5c4442c334f4eb5a2adf4bddb0998: Deleting consensus metadata
I20260925 10:30:30.337309 32690 catalog_manager.cc:4841] TS f6c5c4442c334f4eb5a2adf4bddb0998 (127.12.248.66:36065): tablet 30faee43076e4fbbb3afc94183079cda (table test-table [id=7c988671e6c340988528803a01c4498e]) successfully deleted
I20260925 10:30:30.340113 741 ts_tablet_manager.cc:1918] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:30.340309 741 log.cc:1196] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/30faee43076e4fbbb3afc94183079cda
I20260925 10:30:30.340848 741 ts_tablet_manager.cc:1939] T 30faee43076e4fbbb3afc94183079cda P 5f93d834f6564727bb36524a7adacb07: Deleting consensus metadata
I20260925 10:30:30.329409 740 log.cc:1196] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/30faee43076e4fbbb3afc94183079cda
I20260925 10:30:30.342286 740 ts_tablet_manager.cc:1939] T 30faee43076e4fbbb3afc94183079cda P 479b0e4a94d54dbcaace5812ba447cc6: Deleting consensus metadata
I20260925 10:30:30.342967 32691 catalog_manager.cc:4841] TS 5f93d834f6564727bb36524a7adacb07 (127.12.248.67:38309): tablet 30faee43076e4fbbb3afc94183079cda (table test-table [id=7c988671e6c340988528803a01c4498e]) successfully deleted
I20260925 10:30:30.344568 32689 catalog_manager.cc:4841] TS 479b0e4a94d54dbcaace5812ba447cc6 (127.12.248.65:40881): tablet 30faee43076e4fbbb3afc94183079cda (table test-table [id=7c988671e6c340988528803a01c4498e]) successfully deleted
I20260925 10:30:30.346246 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.307860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.12.248.65:40881)
0925 10:30:30.308098 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.308107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.322783 (+ 14676us) server_negotiation.cc:407] Connection header received
0925 10:30:30.323085 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.323091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.323173 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.323252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:30.325135 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.326341 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.328290 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.328983 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.329084 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.335877 (+ 6793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:30.335898 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:30.335920 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:30.335992 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:30.341484 (+ 5492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.342799 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.342809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.342812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.343016 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.343639 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.343644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.343647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.344070 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:30.344448 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.345315 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.345655 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 10:30:30.347200 422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:30.438928 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32766
I20260925 10:30:30.457943 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 433
Sep 25 10:30:30 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.66@KRBTEST.COM
I20260925 10:30:30.477264 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.458786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.12.248.66:36065)
0925 10:30:30.458961 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.458965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.459055 (+ 90us) server_negotiation.cc:407] Connection header received
0925 10:30:30.459261 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.459265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.459339 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.459426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:30.461260 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.462283 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.464416 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.464870 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.464970 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.470469 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:30.470484 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:30.470501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:30.470540 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:30.473940 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.474689 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.474697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.474700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.474906 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.475315 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.475322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.475325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.475737 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:30.476274 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.476717 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.476909 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:30:30.478013 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:30.593197 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 433
I20260925 10:30:30.612623 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 567
Sep 25 10:30:30 dist-test-slave-f9rl krb5kdc[32659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.67@KRBTEST.COM
I20260925 10:30:30.628175 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:30.613497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.12.248.67:38309)
0925 10:30:30.613689 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:30.613694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:30.613713 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:30:30.613817 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:30.613820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:30.613884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:30.613968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:30.615320 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.616065 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.617550 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:30.617915 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:30.618007 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:30.622948 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:30.622965 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:30.622982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:30.623022 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:30.625722 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.626413 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.626420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.626422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.626569 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:30.626921 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:30.626924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:30.626926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:30.627286 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:30.627499 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:30.627750 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 10:30:30.627921 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:30:30.628887 691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:30.720239 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 567
I20260925 10:30:30.737169 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 32672
I20260925 10:30:30.738358 32735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:30.850241 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 32672
2026-09-25T10:30:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4618 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:30Z Disabled control of system clock
I20260925 10:30:30.929916 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37909
--webserver_interface=127.12.248.126
--webserver_port=0
--builtin_ntp_servers=127.12.248.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:31.274632 751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:31.274940 751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:31.275105 751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:31.282333 751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:31.282439 751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:31.282534 751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:31.282642 751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:31.282725 751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:31.292044 751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:31.293525 751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:31.295305 751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:31.307039 756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:31.306988 757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:31.307018 759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:31.307971 751 server_base.cc:1034] running on GCE node
I20260925 10:30:31.308769 751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:31.310230 751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:31.311465 751 hybrid_clock.cc:648] HybridClock initialized: now 1790332231311429 us; error 50 us; skew 500 ppm
I20260925 10:30:31.311820 751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:31.314108 751 webserver.cc:460] Webserver started at http://127.12.248.126:42225/ using document root <none> and password file <none>
I20260925 10:30:31.314824 751 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:31.314960 751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:31.315290 751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:31.318310 751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "b3fb3ddf6c3247579f69dc1b97a80900"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "197212640418f8517cd06be2b09b32cc"
server_key_iv: "213bf5e163b3a842a39319c89fcf18e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.319101 751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "b3fb3ddf6c3247579f69dc1b97a80900"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "197212640418f8517cd06be2b09b32cc"
server_key_iv: "213bf5e163b3a842a39319c89fcf18e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.324318 751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:30:31.327870 765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:31.329213 751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:30:31.329401 751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "b3fb3ddf6c3247579f69dc1b97a80900"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "197212640418f8517cd06be2b09b32cc"
server_key_iv: "213bf5e163b3a842a39319c89fcf18e7"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.329639 751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:31.348541 751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:31.349422 751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:31.349838 751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:31.369649 751 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37909
I20260925 10:30:31.369707 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37909 every 8 connection(s)
I20260925 10:30:31.371254 751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:31.374713 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 751
I20260925 10:30:31.374931 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:31.375341 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:31.375429 13281 external_mini_cluster.cc:1428] Setting key 3358384e2e32d27b56fa41c89ab118e6
I20260925 10:30:31.388255 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: Bootstrap starting.
I20260925 10:30:31.388803 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:31.377273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.12.248.126:37909)
0925 10:30:31.378310 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:31.378321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:31.378368 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:30:31.380110 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:31.380132 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:31.380603 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:31.381006 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:31.381629 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.382926 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.384349 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.385041 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.385266 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:31.386494 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:31.386502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:31.386514 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:31.387047 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:31.387128 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:31.387149 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:31.387293 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:31.387519 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 10:30:31.387825 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":380,"threads_started":1}
I20260925 10:30:31.391963 817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:31.392961 817 log.cc:824] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:31.395586 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: No bootstrap required, opened a new log
I20260925 10:30:31.403488 817 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } }
I20260925 10:30:31.403997 817 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:31.404130 817 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fb3ddf6c3247579f69dc1b97a80900, State: Initialized, Role: FOLLOWER
I20260925 10:30:31.404786 817 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } }
I20260925 10:30:31.405025 817 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:31.405157 817 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:31.405337 817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:31.407656 817 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } }
I20260925 10:30:31.408213 817 leader_election.cc:302] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3fb3ddf6c3247579f69dc1b97a80900; no voters:
I20260925 10:30:31.408806 817 leader_election.cc:288] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:31.408994 822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:31.409889 822 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [term 1 LEADER]: Becoming Leader. State: Replica: b3fb3ddf6c3247579f69dc1b97a80900, State: Running, Role: LEADER
I20260925 10:30:31.410445 822 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } }
I20260925 10:30:31.411036 817 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:31.413358 824 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3fb3ddf6c3247579f69dc1b97a80900. Latest consensus state: current_term: 1 leader_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } } }
I20260925 10:30:31.413950 824 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:31.413364 823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3fb3ddf6c3247579f69dc1b97a80900" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37909 } } }
I20260925 10:30:31.414567 823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:31.416038 828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:31.422173 828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:31.431303 828 catalog_manager.cc:1347] Generated new cluster ID: e6dd3b47777d4304a307dc3631969f60
I20260925 10:30:31.431430 828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:31.461380 828 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:31.462775 828 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:31.480674 828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: Generated new TSK 0
I20260925 10:30:31.481664 828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:31.501725 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.65:0
--local_ip_for_outbound_sockets=127.12.248.65
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--builtin_ntp_servers=127.12.248.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:30:31.852031 841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:31.852365 841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:31.852521 841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:31.860044 841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:31.860158 841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:31.860345 841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.65
I20260925 10:30:31.869808 841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.248.65
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.65
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:31.871183 841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:31.873037 841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:31.887065 849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:31.887029 847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:31.887087 846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:31.888088 841 server_base.cc:1034] running on GCE node
I20260925 10:30:31.888779 841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:31.889906 841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:31.891135 841 hybrid_clock.cc:648] HybridClock initialized: now 1790332231891087 us; error 74 us; skew 500 ppm
I20260925 10:30:31.891479 841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:31.893918 841 webserver.cc:460] Webserver started at http://127.12.248.65:43225/ using document root <none> and password file <none>
I20260925 10:30:31.894660 841 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:31.894804 841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:31.895116 841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:31.898201 841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data/instance:
uuid: "a575aef9b00748049af348b8eb0b7639"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "8b2c4af0de3a7598b14e302051149ecc"
server_key_iv: "c4d0ee9e898defe7fa8bf9d00e6ea934"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.899000 841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance:
uuid: "a575aef9b00748049af348b8eb0b7639"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "8b2c4af0de3a7598b14e302051149ecc"
server_key_iv: "c4d0ee9e898defe7fa8bf9d00e6ea934"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.904475 841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:30:31.908138 855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:31.909644 841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:31.909825 841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal
uuid: "a575aef9b00748049af348b8eb0b7639"
format_stamp: "Formatted at 2026-09-25 10:30:31 on dist-test-slave-f9rl"
server_key: "8b2c4af0de3a7598b14e302051149ecc"
server_key_iv: "c4d0ee9e898defe7fa8bf9d00e6ea934"
server_key_version: "encryptionkey@0"
I20260925 10:30:31.910055 841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:31.932304 841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:31.933243 841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:31.933657 841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:31.934679 841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:31.936239 841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:31.936323 841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:31.936478 841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:31.936579 841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:31.970887 841 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.65:43457
I20260925 10:30:31.970992 969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.65:43457 every 8 connection(s)
I20260925 10:30:31.972244 841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/data/info.pb
I20260925 10:30:31.981819 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 841
I20260925 10:30:31.982002 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal/instance
I20260925 10:30:31.982455 13281 external_mini_cluster.cc:1428] Setting key a10660daf4105fb29b641a0a7b3eb4e6
I20260925 10:30:31.985824 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.66:0
--local_ip_for_outbound_sockets=127.12.248.66
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--builtin_ntp_servers=127.12.248.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:31.985954 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:31.975090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:36173 (local address 127.12.248.126:37909)
0925 10:30:31.975544 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:31.975560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:31.976787 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 10:30:31.978651 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:31.978655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:31.978725 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:31.978826 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:31.979936 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.980882 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.982701 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.983274 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.983428 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:31.985093 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:31.985098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:31.985101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:31.985120 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:31.985160 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:31.985171 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:31.985304 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:31.985577 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:30:31.985657 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260925 10:30:31.987591 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:31.975698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37909 (local address 127.12.248.65:36173)
0925 10:30:31.976604 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:31.976646 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:31.978389 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:31.979031 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:31.979039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:31.979125 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:31.979752 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:31.979771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.981034 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.981037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:31.982538 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:31.982555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:31.983524 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:31.983529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:31.983860 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:31.984725 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:31.984728 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:31.984784 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:31.984896 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:31.985302 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:31.985314 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:31.985323 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:30:31.985550 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 10:30:31.986008 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260925 10:30:31.989148 970 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37909
I20260925 10:30:31.989514 970 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:31.990266 970 heartbeater.cc:511] Master 127.12.248.126:37909 requested a full tablet report, sending...
I20260925 10:30:31.992471 782 ts_manager.cc:194] Registered new tserver with Master: a575aef9b00748049af348b8eb0b7639 (127.12.248.65:43457)
I20260925 10:30:31.994273 782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.65:36173
W20260925 10:30:32.326980 974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:32.327273 974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:32.327360 974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:32.334743 974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:32.334851 974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:32.334959 974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.66
I20260925 10:30:32.344099 974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.248.66
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.66
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:32.345331 974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:32.347065 974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:32.364439 980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:32.364495 979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:32.364375 982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:32.365523 974 server_base.cc:1034] running on GCE node
I20260925 10:30:32.366195 974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:32.367259 974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:32.368543 974 hybrid_clock.cc:648] HybridClock initialized: now 1790332232368420 us; error 95 us; skew 500 ppm
I20260925 10:30:32.369006 974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:32.371994 974 webserver.cc:460] Webserver started at http://127.12.248.66:39153/ using document root <none> and password file <none>
I20260925 10:30:32.372920 974 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:32.373036 974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:32.373346 974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:32.377337 974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data/instance:
uuid: "ae5cd7f54d454948b23846ac71776184"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "3c6a0c810f14813a02e83f73edd88b90"
server_key_iv: "74bb6e26d306fc7323f44299ed7b8a1b"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.378382 974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5cd7f54d454948b23846ac71776184"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "3c6a0c810f14813a02e83f73edd88b90"
server_key_iv: "74bb6e26d306fc7323f44299ed7b8a1b"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.385028 974 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 10:30:32.389482 988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.391039 974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:32.391208 974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal
uuid: "ae5cd7f54d454948b23846ac71776184"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "3c6a0c810f14813a02e83f73edd88b90"
server_key_iv: "74bb6e26d306fc7323f44299ed7b8a1b"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.391426 974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:32.406985 974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:32.407804 974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:32.408114 974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:32.409104 974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:32.410640 974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:32.410734 974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.410889 974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:32.410996 974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.446447 974 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.66:36561
I20260925 10:30:32.446470 1100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.66:36561 every 8 connection(s)
I20260925 10:30:32.447858 974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/data/info.pb
I20260925 10:30:32.453665 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 974
I20260925 10:30:32.453859 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal/instance
I20260925 10:30:32.454442 13281 external_mini_cluster.cc:1428] Setting key 164026ab253eab1028c21559c7f2a1ba
I20260925 10:30:32.457549 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.248.67:0
--local_ip_for_outbound_sockets=127.12.248.67
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--builtin_ntp_servers=127.12.248.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:30:32.461498 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.450833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.66:54889 (local address 127.12.248.126:37909)
0925 10:30:32.451057 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.451064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.452403 (+ 1339us) server_negotiation.cc:407] Connection header received
0925 10:30:32.454746 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.454750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.454824 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.454913 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:32.456223 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.457029 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.458750 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.459139 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.459282 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:32.460652 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:32.460657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:32.460660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:32.460681 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:32.460715 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:32.460727 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:32.460819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:32.461044 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:30:32.461145 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:30:32.462517 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.451294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37909 (local address 127.12.248.66:54889)
0925 10:30:32.452225 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:32.452261 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:32.454412 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:32.455187 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:32.455195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:32.455285 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:32.456041 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.456064 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.457175 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.457183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.458567 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.458579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.459370 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.459376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.459605 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:32.460332 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:32.460335 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:32.460374 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:32.460451 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:32.460833 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:32.460846 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:32.460860 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:30:32.461036 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 10:30:32.461429 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260925 10:30:32.463966 1101 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37909
I20260925 10:30:32.464304 1101 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:32.465152 1101 heartbeater.cc:511] Master 127.12.248.126:37909 requested a full tablet report, sending...
I20260925 10:30:32.466674 782 ts_manager.cc:194] Registered new tserver with Master: ae5cd7f54d454948b23846ac71776184 (127.12.248.66:36561)
I20260925 10:30:32.467471 782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.66:54889
W20260925 10:30:32.785203 1104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:32.785565 1104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:32.785723 1104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:32.793085 1104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:32.793241 1104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:32.793427 1104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.248.67
I20260925 10:30:32.802651 1104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.248.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.248.67
--webserver_port=0
--tserver_master_addrs=127.12.248.126:37909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.248.67
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:32.803989 1104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:32.805795 1104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:32.819424 1112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:32.819377 1110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:32.819468 1109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:32.823202 1104 server_base.cc:1034] running on GCE node
I20260925 10:30:32.823899 1104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:32.824913 1104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:32.826145 1104 hybrid_clock.cc:648] HybridClock initialized: now 1790332232826110 us; error 46 us; skew 500 ppm
I20260925 10:30:32.826502 1104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 10:30:32.828877 1104 webserver.cc:460] Webserver started at http://127.12.248.67:39063/ using document root <none> and password file <none>
I20260925 10:30:32.829699 1104 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:32.829835 1104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:32.830164 1104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:32.833184 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data/instance:
uuid: "2659fba249fb4ef3836af4470f978f5d"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "4d929d4e61b62e0f6dd67c06147f6515"
server_key_iv: "05727ad7888da72b0202e46d665bac79"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.834077 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance:
uuid: "2659fba249fb4ef3836af4470f978f5d"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "4d929d4e61b62e0f6dd67c06147f6515"
server_key_iv: "05727ad7888da72b0202e46d665bac79"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.839443 1104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:32.843137 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.844509 1104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:30:32.844694 1104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal
uuid: "2659fba249fb4ef3836af4470f978f5d"
format_stamp: "Formatted at 2026-09-25 10:30:32 on dist-test-slave-f9rl"
server_key: "4d929d4e61b62e0f6dd67c06147f6515"
server_key_iv: "05727ad7888da72b0202e46d665bac79"
server_key_version: "encryptionkey@0"
I20260925 10:30:32.844945 1104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:32.868803 1104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:32.869773 1104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:32.870090 1104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:32.871125 1104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:30:32.872643 1104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:30:32.872774 1104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.872900 1104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:30:32.872987 1104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:32.907371 1104 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.67:32817
I20260925 10:30:32.907462 1230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.67:32817 every 8 connection(s)
I20260925 10:30:32.908695 1104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/data/info.pb
I20260925 10:30:32.913712 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1104
I20260925 10:30:32.913890 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal/instance
I20260925 10:30:32.914448 13281 external_mini_cluster.cc:1428] Setting key 67b8b7644b9c042547fc562c3e554f3f
I20260925 10:30:32.922180 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.911760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.67:53691 (local address 127.12.248.126:37909)
0925 10:30:32.911991 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.911996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.913305 (+ 1309us) server_negotiation.cc:407] Connection header received
0925 10:30:32.915479 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.915484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.915584 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.915694 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:32.916934 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.917893 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.919452 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.919856 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.919984 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:32.921296 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:32.921301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:32.921308 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:32.921333 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:32.921365 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:32.921377 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:32.921512 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:32.921739 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:30:32.921836 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 10:30:32.923215 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.912261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:37909 (local address 127.12.248.67:53691)
0925 10:30:32.913145 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:32.913187 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:32.915161 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:32.915965 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:32.915976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:32.916093 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:30:32.916772 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.916789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.918050 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.918055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.919303 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.919315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.920085 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.920092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.920332 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:32.920981 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:30:32.920984 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:30:32.921023 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:30:32.921105 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:30:32.921502 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:30:32.921512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:30:32.921521 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:30:32.921731 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 10:30:32.922134 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260925 10:30:32.924556 1231 heartbeater.cc:347] Connected to a master server at 127.12.248.126:37909
I20260925 10:30:32.924840 1231 heartbeater.cc:464] Registering TS with master...
I20260925 10:30:32.925648 1231 heartbeater.cc:511] Master 127.12.248.126:37909 requested a full tablet report, sending...
I20260925 10:30:32.927271 782 ts_manager.cc:194] Registered new tserver with Master: 2659fba249fb4ef3836af4470f978f5d (127.12.248.67:32817)
I20260925 10:30:32.928485 782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.248.67:53691
I20260925 10:30:32.929926 13281 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:30:32.945580 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.934675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.12.248.126:37909)
0925 10:30:32.934874 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.934879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.935380 (+ 501us) server_negotiation.cc:407] Connection header received
0925 10:30:32.935518 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.935522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.935616 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.935710 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:32.936220 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.936977 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.938378 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.938834 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.938935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:32.944898 (+ 5963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:32.944904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:32.944907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:32.944928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:32.944967 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:32.944979 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:32.945074 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:32.945172 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:30:32.945262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 10:30:32.949914 782 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:30:32.956751 782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:30:32.997977 970 heartbeater.cc:503] Master 127.12.248.126:37909 was elected leader, sending a full tablet report...
I20260925 10:30:33.001340 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.970167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:36561 (local address 127.0.0.1:59914)
0925 10:30:32.992700 (+ 22533us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:32.992718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:32.992856 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:32.993617 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:32.993621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:32.993645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:32.993968 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.993977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.996416 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.996426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.998121 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.998135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.000231 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.000237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.000521 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000635 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 10:30:33.000752 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:30:33.000949 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22385,"mutex_wait_us":73,"thread_start_us":19586,"threads_started":1}
I20260925 10:30:33.001346 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.969717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.12.248.65:43457)
0925 10:30:32.984544 (+ 14827us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.984554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.990547 (+ 5993us) server_negotiation.cc:407] Connection header received
0925 10:30:32.991047 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.991060 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.991402 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.991620 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:32.992980 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.994703 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.998113 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.999643 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.999776 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000070 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.000831 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.000991 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14610,"thread_start_us":14473,"threads_started":1}
I20260925 10:30:33.001415 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.977737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.12.248.67:32817)
0925 10:30:32.988898 (+ 11161us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.988911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.991283 (+ 2372us) server_negotiation.cc:407] Connection header received
0925 10:30:32.991578 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.991600 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.991827 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.991994 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:32.993147 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.994740 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.998768 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.000196 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.000307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000561 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.000893 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.001054 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10898,"thread_start_us":10706,"threads_started":1}
I20260925 10:30:33.001837 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.969789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32817 (local address 127.0.0.1:60298)
0925 10:30:32.991139 (+ 21350us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:32.991157 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:32.991395 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:32.992337 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:32.992341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:32.992377 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:32.992955 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.992968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.994898 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.994904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.997853 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.997861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.000442 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.000448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.000649 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000674 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:30:33.000752 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:30:33.001568 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21213,"mutex_wait_us":845,"thread_start_us":21061,"threads_started":1}
I20260925 10:30:33.001379 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.969641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.65:43457 (local address 127.0.0.1:48746)
0925 10:30:32.990344 (+ 20703us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:32.990401 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:32.990664 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:32.992038 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:32.992048 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:32.992099 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:32.992702 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.992714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.994873 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.994877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:32.997846 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:32.997861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.000283 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.000291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.000564 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000635 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 10:30:33.000776 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 10:30:33.000945 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20312,"mutex_wait_us":328,"thread_start_us":22752,"threads_started":1}
I20260925 10:30:33.002306 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:32.970675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.12.248.66:36561)
0925 10:30:32.989503 (+ 18828us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:32.989524 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:32.992851 (+ 3327us) server_negotiation.cc:407] Connection header received
0925 10:30:32.993093 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:32.993101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:32.993350 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:32.993619 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:32.994135 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.996276 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.998340 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:32.999715 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:32.999839 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.000083 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.001767 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.001929 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18613,"thread_start_us":18501,"threads_started":1}
I20260925 10:30:33.005666 1036 tablet_service.cc:1467] Processing CreateTablet for tablet 927759d347c3449c9533d5a99678bd53 (DEFAULT_TABLE table=test-table [id=6ab122a5335f4d40ab161092fe47a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:33.005690 1166 tablet_service.cc:1467] Processing CreateTablet for tablet 927759d347c3449c9533d5a99678bd53 (DEFAULT_TABLE table=test-table [id=6ab122a5335f4d40ab161092fe47a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:33.006870 1036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927759d347c3449c9533d5a99678bd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:33.006882 1166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927759d347c3449c9533d5a99678bd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:33.011137 904 tablet_service.cc:1467] Processing CreateTablet for tablet 927759d347c3449c9533d5a99678bd53 (DEFAULT_TABLE table=test-table [id=6ab122a5335f4d40ab161092fe47a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:30:33.012645 904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927759d347c3449c9533d5a99678bd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:33.019235 1249 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Bootstrap starting.
I20260925 10:30:33.021479 1250 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Bootstrap starting.
I20260925 10:30:33.024085 1249 tablet_bootstrap.cc:654] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:33.025277 1250 tablet_bootstrap.cc:654] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:33.025380 1249 log.cc:824] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:33.026683 1251 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Bootstrap starting.
I20260925 10:30:33.026681 1250 log.cc:824] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:33.029938 1249 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: No bootstrap required, opened a new log
I20260925 10:30:33.030251 1249 ts_tablet_manager.cc:1397] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 10:30:33.030284 1250 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: No bootstrap required, opened a new log
I20260925 10:30:33.030324 1251 tablet_bootstrap.cc:654] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:33.030535 1250 ts_tablet_manager.cc:1397] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 10:30:33.031538 1251 log.cc:824] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:33.034660 1251 tablet_bootstrap.cc:492] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: No bootstrap required, opened a new log
I20260925 10:30:33.034931 1251 ts_tablet_manager.cc:1397] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 10:30:33.038843 1249 raft_consensus.cc:348] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.039235 1249 raft_consensus.cc:374] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:33.039343 1249 raft_consensus.cc:729] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2659fba249fb4ef3836af4470f978f5d, State: Initialized, Role: FOLLOWER
I20260925 10:30:33.039968 1249 consensus_queue.cc:260] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.041092 1250 raft_consensus.cc:348] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.041576 1250 raft_consensus.cc:374] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:33.041666 1250 raft_consensus.cc:729] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5cd7f54d454948b23846ac71776184, State: Initialized, Role: FOLLOWER
I20260925 10:30:33.042366 1250 consensus_queue.cc:260] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.046461 1251 raft_consensus.cc:348] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.047034 1251 raft_consensus.cc:374] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:33.047147 1251 raft_consensus.cc:729] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a575aef9b00748049af348b8eb0b7639, State: Initialized, Role: FOLLOWER
I20260925 10:30:33.047962 1251 consensus_queue.cc:260] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.048436 1231 heartbeater.cc:503] Master 127.12.248.126:37909 was elected leader, sending a full tablet report...
I20260925 10:30:33.048862 1249 ts_tablet_manager.cc:1428] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Time spent starting tablet: real 0.018s user 0.008s sys 0.009s
I20260925 10:30:33.050443 1250 ts_tablet_manager.cc:1428] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Time spent starting tablet: real 0.020s user 0.014s sys 0.007s
I20260925 10:30:33.050585 1251 ts_tablet_manager.cc:1428] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260925 10:30:33.051057 1101 heartbeater.cc:503] Master 127.12.248.126:37909 was elected leader, sending a full tablet report...
I20260925 10:30:33.055730 1257 raft_consensus.cc:482] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:30:33.056003 1257 raft_consensus.cc:504] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.057605 1257 leader_election.cc:288] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2659fba249fb4ef3836af4470f978f5d (127.12.248.67:32817), ae5cd7f54d454948b23846ac71776184 (127.12.248.66:36561)
I20260925 10:30:33.064951 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.058026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:42011 (local address 127.12.248.66:36561)
0925 10:30:33.058226 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.058232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.059070 (+ 838us) server_negotiation.cc:407] Connection header received
0925 10:30:33.059277 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.059282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.059380 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.059489 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:33.060316 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.061044 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.063205 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.063995 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.064124 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.064203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.064545 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.064680 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:30:33.064950 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.058130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.248.65:36007 (local address 127.12.248.67:32817)
0925 10:30:33.058303 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.058309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.059093 (+ 784us) server_negotiation.cc:407] Connection header received
0925 10:30:33.059324 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.059330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.059400 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.059485 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:30:33.060400 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.061143 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.063273 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.064109 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.064274 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.064344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.064599 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.064724 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:30:33.065090 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.057874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.66:36561 (local address 127.12.248.65:42011)
0925 10:30:33.058941 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:33.058961 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:33.059120 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:33.059510 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:33.059516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:33.059543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:33.060032 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:33.060055 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.061180 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.061188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.062986 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:33.063001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.064183 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.064191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.064367 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.064399 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:30:33.064484 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:30:33.064634 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":131,"thread_start_us":817,"threads_started":1}
I20260925 10:30:33.065120 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.057998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.67:32817 (local address 127.12.248.65:36007)
0925 10:30:33.059002 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 10:30:33.059023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:30:33.059171 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:30:33.059653 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:30:33.059658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:30:33.059697 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:30:33.060160 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:33.060184 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.061289 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.061296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.063029 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:30:33.063046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.064306 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.064311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:30:33.064424 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.064445 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 10:30:33.064565 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:30:33.064685 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":144,"thread_start_us":449,"threads_started":1}
I20260925 10:30:33.066112 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927759d347c3449c9533d5a99678bd53" candidate_uuid: "a575aef9b00748049af348b8eb0b7639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2659fba249fb4ef3836af4470f978f5d" is_pre_election: true
I20260925 10:30:33.066340 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927759d347c3449c9533d5a99678bd53" candidate_uuid: "a575aef9b00748049af348b8eb0b7639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5cd7f54d454948b23846ac71776184" is_pre_election: true
I20260925 10:30:33.066725 1186 raft_consensus.cc:2454] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a575aef9b00748049af348b8eb0b7639 in term 0.
I20260925 10:30:33.066877 1056 raft_consensus.cc:2454] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a575aef9b00748049af348b8eb0b7639 in term 0.
I20260925 10:30:33.067426 857 leader_election.cc:302] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2659fba249fb4ef3836af4470f978f5d, a575aef9b00748049af348b8eb0b7639; no voters:
I20260925 10:30:33.067830 1257 raft_consensus.cc:2781] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:30:33.068001 1257 raft_consensus.cc:482] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:30:33.068133 1257 raft_consensus.cc:3037] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:33.070502 1257 raft_consensus.cc:504] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.071118 1257 leader_election.cc:288] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [CANDIDATE]: Term 1 election: Requested vote from peers 2659fba249fb4ef3836af4470f978f5d (127.12.248.67:32817), ae5cd7f54d454948b23846ac71776184 (127.12.248.66:36561)
I20260925 10:30:33.071630 1056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927759d347c3449c9533d5a99678bd53" candidate_uuid: "a575aef9b00748049af348b8eb0b7639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5cd7f54d454948b23846ac71776184"
I20260925 10:30:33.071645 1186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927759d347c3449c9533d5a99678bd53" candidate_uuid: "a575aef9b00748049af348b8eb0b7639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2659fba249fb4ef3836af4470f978f5d"
I20260925 10:30:33.071851 1186 raft_consensus.cc:3037] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:33.071828 1056 raft_consensus.cc:3037] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:33.074203 1056 raft_consensus.cc:2454] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a575aef9b00748049af348b8eb0b7639 in term 1.
I20260925 10:30:33.074206 1186 raft_consensus.cc:2454] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a575aef9b00748049af348b8eb0b7639 in term 1.
I20260925 10:30:33.074653 859 leader_election.cc:302] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a575aef9b00748049af348b8eb0b7639, ae5cd7f54d454948b23846ac71776184; no voters:
I20260925 10:30:33.074939 1257 raft_consensus.cc:2781] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:33.075338 1257 raft_consensus.cc:686] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 1 LEADER]: Becoming Leader. State: Replica: a575aef9b00748049af348b8eb0b7639, State: Running, Role: LEADER
I20260925 10:30:33.075791 1257 consensus_queue.cc:237] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } }
I20260925 10:30:33.080874 782 catalog_manager.cc:5495] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 reported cstate change: term changed from 0 to 1, leader changed from <none> to a575aef9b00748049af348b8eb0b7639 (127.12.248.65). New cstate: current_term: 1 leader_uuid: "a575aef9b00748049af348b8eb0b7639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a575aef9b00748049af348b8eb0b7639" member_type: VOTER last_known_addr { host: "127.12.248.65" port: 43457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 } health_report { overall_health: UNKNOWN } } }
I20260925 10:30:33.110168 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.104387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.12.248.65:43457)
0925 10:30:33.104637 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.104643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.104728 (+ 85us) server_negotiation.cc:407] Connection header received
0925 10:30:33.105014 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.105018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.105117 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.105284 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:30:33.105700 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.106405 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.107889 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.108315 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.108453 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.108967 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:30:33.109586 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:30:33.109760 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.109812 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.109896 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:30:33.119158 1186 raft_consensus.cc:1261] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 1 FOLLOWER]: Refusing update from remote peer a575aef9b00748049af348b8eb0b7639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:33.119158 1056 raft_consensus.cc:1261] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 1 FOLLOWER]: Refusing update from remote peer a575aef9b00748049af348b8eb0b7639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:30:33.120224 1260 consensus_queue.cc:1035] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2659fba249fb4ef3836af4470f978f5d" member_type: VOTER last_known_addr { host: "127.12.248.67" port: 32817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:33.120591 1257 consensus_queue.cc:1035] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5cd7f54d454948b23846ac71776184" member_type: VOTER last_known_addr { host: "127.12.248.66" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:30:33.135773 1265 mvcc.cc:205] Tried to move back new op lower bound from 7333200826844606464 to 7333200826682609664. Current Snapshot: MvccSnapshot[applied={T|T < 7333200826844606464}]
I20260925 10:30:33.149029 782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:33.149379 782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:30:33.153074 782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3fb3ddf6c3247579f69dc1b97a80900: Sending DeleteTablet for 3 replicas of tablet 927759d347c3449c9533d5a99678bd53
I20260925 10:30:33.154273 904 tablet_service.cc:1514] Processing DeleteTablet for tablet 927759d347c3449c9533d5a99678bd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:33 UTC) from {username='slave'} at 127.0.0.1:48746
I20260925 10:30:33.154301 1166 tablet_service.cc:1514] Processing DeleteTablet for tablet 927759d347c3449c9533d5a99678bd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:33 UTC) from {username='slave'} at 127.0.0.1:60298
I20260925 10:30:33.154359 1036 tablet_service.cc:1514] Processing DeleteTablet for tablet 927759d347c3449c9533d5a99678bd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:30:33 UTC) from {username='slave'} at 127.0.0.1:59914
I20260925 10:30:33.158828 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 841
I20260925 10:30:33.166770 1275 tablet_replica.cc:330] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: stopping tablet replica
I20260925 10:30:33.167390 1276 tablet_replica.cc:330] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: stopping tablet replica
I20260925 10:30:33.167822 1276 raft_consensus.cc:2229] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:33.168357 1276 raft_consensus.cc:2258] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:33.169135 1275 raft_consensus.cc:2229] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:30:33.169673 1275 raft_consensus.cc:2258] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:33.170452 1274 tablet_replica.cc:330] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: stopping tablet replica
I20260925 10:30:33.170847 1274 raft_consensus.cc:2229] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:30:33.171041 1276 ts_tablet_manager.cc:1905] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:33.171634 1274 raft_consensus.cc:2258] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:30:33.173379 1275 ts_tablet_manager.cc:1905] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:33.173619 1274 ts_tablet_manager.cc:1905] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:30:33.180835 1276 ts_tablet_manager.cc:1918] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:33.180980 1274 ts_tablet_manager.cc:1918] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:33.181150 1274 log.cc:1196] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-0/wal/wals/927759d347c3449c9533d5a99678bd53
I20260925 10:30:33.181772 1274 ts_tablet_manager.cc:1939] T 927759d347c3449c9533d5a99678bd53 P a575aef9b00748049af348b8eb0b7639: Deleting consensus metadata
I20260925 10:30:33.183841 766 catalog_manager.cc:4841] TS a575aef9b00748049af348b8eb0b7639 (127.12.248.65:43457): tablet 927759d347c3449c9533d5a99678bd53 (table test-table [id=6ab122a5335f4d40ab161092fe47a024]) successfully deleted
I20260925 10:30:33.184679 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.160345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.12.248.65:43457)
0925 10:30:33.161831 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.161838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.175115 (+ 13277us) server_negotiation.cc:407] Connection header received
0925 10:30:33.175635 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.175641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.175746 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.175839 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:33.176504 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.177798 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.179704 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.180378 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.180485 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.182128 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:33.182144 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:33.182169 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:33.183060 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:33.183136 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:33.183162 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:33.183242 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.184196 (+ 954us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.184305 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337}
I20260925 10:30:33.185388 1276 log.cc:1196] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-1/wal/wals/927759d347c3449c9533d5a99678bd53
I20260925 10:30:33.186021 965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:33.187696 1276 ts_tablet_manager.cc:1939] T 927759d347c3449c9533d5a99678bd53 P ae5cd7f54d454948b23846ac71776184: Deleting consensus metadata
I20260925 10:30:33.188767 1275 ts_tablet_manager.cc:1918] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:30:33.188968 1275 log.cc:1196] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Deleting WAL directory at /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331906218802-13281-0/minicluster-data/ts-2/wal/wals/927759d347c3449c9533d5a99678bd53
I20260925 10:30:33.189504 1275 ts_tablet_manager.cc:1939] T 927759d347c3449c9533d5a99678bd53 P 2659fba249fb4ef3836af4470f978f5d: Deleting consensus metadata
I20260925 10:30:33.190143 769 catalog_manager.cc:4841] TS ae5cd7f54d454948b23846ac71776184 (127.12.248.66:36561): tablet 927759d347c3449c9533d5a99678bd53 (table test-table [id=6ab122a5335f4d40ab161092fe47a024]) successfully deleted
I20260925 10:30:33.191704 767 catalog_manager.cc:4841] TS 2659fba249fb4ef3836af4470f978f5d (127.12.248.67:32817): tablet 927759d347c3449c9533d5a99678bd53 (table test-table [id=6ab122a5335f4d40ab161092fe47a024]) successfully deleted
I20260925 10:30:33.284523 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 841
I20260925 10:30:33.303632 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 974
I20260925 10:30:33.310268 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.304422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.12.248.66:36561)
0925 10:30:33.304594 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.304598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.304619 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:30:33.304799 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.304803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.304880 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.304967 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:33.305430 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.306230 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.307445 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.307839 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.307932 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.308906 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:33.308913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:33.308927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:33.309608 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:33.309656 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:33.309677 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:33.309764 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.309890 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.309979 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:30:33.311044 1096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:33.400377 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 974
I20260925 10:30:33.417891 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1104
I20260925 10:30:33.424983 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:33.418709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.12.248.67:32817)
0925 10:30:33.418917 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:33.418923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:33.419027 (+ 104us) server_negotiation.cc:407] Connection header received
0925 10:30:33.419262 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:33.419268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:33.419359 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:33.419465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:33.419859 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.420572 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.421863 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:33.422255 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:33.422384 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:33.423445 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:33.423455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:33.423469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:33.424254 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:33.424303 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:33.424323 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:33.424446 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:33.424581 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:30:33.424697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:30:33.425724 1226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:33.514480 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1104
I20260925 10:30:33.531250 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 751
I20260925 10:30:33.532351 812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:33.642264 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 751
2026-09-25T10:30:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2791 ms)
[----------] 6 tests from EncryptionPolicyTest (22415 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:33 dist-test-slave-f9rl krb5kdc[1284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:33 dist-test-slave-f9rl krb5kdc[1284](info): set up 2 sockets
Sep 25 10:30:33 dist-test-slave-f9rl krb5kdc[1284](info): commencing operation
krb5kdc: starting...
W20260925 10:30:34.787377 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:34 dist-test-slave-f9rl krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:35.055462 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:35.399302 1297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:35.399755 1297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:35.399919 1297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:35.407276 1297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:35.407449 1297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:35.407570 1297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:35.407686 1297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:35.407756 1297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:35.417881 1297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:35.419267 1297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:35.420734 1297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:35.432626 1302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:35.432659 1303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:35.432863 1305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:35.433673 1297 server_base.cc:1034] running on GCE node
I20260925 10:30:35.434435 1297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:35.435842 1297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:35.437042 1297 hybrid_clock.cc:648] HybridClock initialized: now 1790332235436988 us; error 49 us; skew 500 ppm
I20260925 10:30:35.437414 1297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:35 dist-test-slave-f9rl krb5kdc[1284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:35.440714 1297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:35.442278 1297 webserver.cc:460] Webserver started at http://127.0.0.1:42601/ using document root <none> and password file <none>
I20260925 10:30:35.442974 1297 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:35.443111 1297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:35.443464 1297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:35.446797 1297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "6477aa03a10046308b82380dd361b6d8"
format_stamp: "Formatted at 2026-09-25 10:30:35 on dist-test-slave-f9rl"
server_key: "31bff2e319c89de5efeb6c31e38679c3"
server_key_iv: "0cb44d6d8a103bedb646958f02993de5"
server_key_version: "encryptionkey@0"
I20260925 10:30:35.447970 1297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "6477aa03a10046308b82380dd361b6d8"
format_stamp: "Formatted at 2026-09-25 10:30:35 on dist-test-slave-f9rl"
server_key: "31bff2e319c89de5efeb6c31e38679c3"
server_key_iv: "0cb44d6d8a103bedb646958f02993de5"
server_key_version: "encryptionkey@0"
I20260925 10:30:35.455530 1297 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260925 10:30:35.459461 1312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:35.460924 1297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:30:35.461114 1297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "6477aa03a10046308b82380dd361b6d8"
format_stamp: "Formatted at 2026-09-25 10:30:35 on dist-test-slave-f9rl"
server_key: "31bff2e319c89de5efeb6c31e38679c3"
server_key_iv: "0cb44d6d8a103bedb646958f02993de5"
server_key_version: "encryptionkey@0"
I20260925 10:30:35.461331 1297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:35.504673 1297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:35.508993 1297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:35.509305 1297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:35.528849 1297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42849
I20260925 10:30:35.528882 1363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42849 every 8 connection(s)
I20260925 10:30:35.530498 1297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:35.532089 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1297
I20260925 10:30:35.532333 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:35.533216 13281 external_mini_cluster.cc:1428] Setting key 1b95d8c933e2b7cfc5c1461bc9ac53e9
I20260925 10:30:35.535179 1364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:35.546633 1364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8: Bootstrap starting.
Sep 25 10:30:35 dist-test-slave-f9rl krb5kdc[1284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:30:35.549940 1364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:35.551008 1364 log.cc:824] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:35.553848 1364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8: No bootstrap required, opened a new log
I20260925 10:30:35.555948 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:35.535370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46241 (local address 127.0.0.1:42849)
0925 10:30:35.536165 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:35.536177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:35.536218 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:30:35.537987 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:35.538009 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:35.538455 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:35.538885 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:35.540369 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:35.541756 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:35.542991 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:35.543669 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:35.543880 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:35.548459 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:35.548467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:35.548481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:35.548529 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:35.551744 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:35.552548 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:35.552556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:35.552563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:35.552740 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:35.553227 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:35.553238 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:35.553241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:35.553799 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:35.554125 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:35.554599 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 10:30:35.554905 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":350,"threads_started":1}
I20260925 10:30:35.560143 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:35.562263 1364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260925 10:30:35.562670 1364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:35.562816 1364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6477aa03a10046308b82380dd361b6d8, State: Initialized, Role: FOLLOWER
I20260925 10:30:35.563598 1364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260925 10:30:35.563925 1364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:35.564067 1364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:35.564254 1364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:35.568929 1364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260925 10:30:35.569710 1364 leader_election.cc:302] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6477aa03a10046308b82380dd361b6d8; no voters:
I20260925 10:30:35.570430 1364 leader_election.cc:288] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:35.570550 1374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:35.571245 1374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6477aa03a10046308b82380dd361b6d8, State: Running, Role: LEADER
I20260925 10:30:35.572029 1374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260925 10:30:35.575428 1364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:35.583436 1375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6477aa03a10046308b82380dd361b6d8. Latest consensus state: current_term: 1 leader_uuid: "6477aa03a10046308b82380dd361b6d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } } }
I20260925 10:30:35.584211 1375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:35.583779 1376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6477aa03a10046308b82380dd361b6d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6477aa03a10046308b82380dd361b6d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } } }
I20260925 10:30:35.584936 1376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:35.589986 1383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:35.592288 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:35.565217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53703 (local address 127.0.0.1:42849)
0925 10:30:35.565397 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:35.565401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:35.569845 (+ 4444us) server_negotiation.cc:407] Connection header received
0925 10:30:35.570107 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:35.570117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:35.570234 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:35.570353 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:35.572660 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:35.574013 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:35.575816 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:35.576804 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:35.576969 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:35.581147 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:35.581157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:35.581165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:35.581227 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:35.588587 (+ 7360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:35.589827 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:35.589837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:35.589840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:35.590070 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:35.590611 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:35.590614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:35.590616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:35.590955 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:35.591161 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:35.591624 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 10:30:35.591785 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:30:35.596623 1383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:35.606050 1383 catalog_manager.cc:1347] Generated new cluster ID: dd9465a22fa34456a324e448a8657cb4
I20260925 10:30:35.606191 1383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:35.622066 1383 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:35.623402 1383 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:35.633271 1383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6477aa03a10046308b82380dd361b6d8: Generated new TSK 0
I20260925 10:30:35.634171 1383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:35.643846 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1297
I20260925 10:30:35.645926 1359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:35.738770 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1297
W20260925 10:30:35.756785 1282 connection.cc:535] client connection to 127.0.0.1:42849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42849): Connection reset by peer (error 104)
2026-09-25T10:30:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:35 dist-test-slave-f9rl krb5kdc[1397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:35 dist-test-slave-f9rl krb5kdc[1397](info): set up 2 sockets
Sep 25 10:30:35 dist-test-slave-f9rl krb5kdc[1397](info): commencing operation
krb5kdc: starting...
W20260925 10:30:36.923295 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:37 dist-test-slave-f9rl krb5kdc[1397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:37.196265 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:37.530119 1410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:37.530417 1410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:37.530511 1410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:37.537750 1410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:37.537858 1410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:37.537907 1410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:37.537967 1410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:37.538003 1410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:37.547639 1410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:37.548894 1410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:37.550170 1410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:37.562475 1415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:37.562454 1416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:37.562640 1418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:37.563326 1410 server_base.cc:1034] running on GCE node
I20260925 10:30:37.564159 1410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:37.565811 1410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:37.567061 1410 hybrid_clock.cc:648] HybridClock initialized: now 1790332237567020 us; error 34 us; skew 500 ppm
I20260925 10:30:37.567466 1410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:37 dist-test-slave-f9rl krb5kdc[1397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:37.570842 1410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:37.572309 1410 webserver.cc:460] Webserver started at http://127.0.0.1:39387/ using document root <none> and password file <none>
I20260925 10:30:37.572974 1410 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:37.573079 1410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:37.573336 1410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:37.576468 1410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "48084dc10b434b559d2480c82b635224"
format_stamp: "Formatted at 2026-09-25 10:30:37 on dist-test-slave-f9rl"
server_key: "c05d3c653b1d8e10680d1c869c8f488a"
server_key_iv: "b30bb60fcee113140c9a0ec6c70779e6"
server_key_version: "encryptionkey@0"
I20260925 10:30:37.577239 1410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "48084dc10b434b559d2480c82b635224"
format_stamp: "Formatted at 2026-09-25 10:30:37 on dist-test-slave-f9rl"
server_key: "c05d3c653b1d8e10680d1c869c8f488a"
server_key_iv: "b30bb60fcee113140c9a0ec6c70779e6"
server_key_version: "encryptionkey@0"
I20260925 10:30:37.582791 1410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:30:37.586519 1425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:37.588181 1410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:37.588366 1410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "48084dc10b434b559d2480c82b635224"
format_stamp: "Formatted at 2026-09-25 10:30:37 on dist-test-slave-f9rl"
server_key: "c05d3c653b1d8e10680d1c869c8f488a"
server_key_iv: "b30bb60fcee113140c9a0ec6c70779e6"
server_key_version: "encryptionkey@0"
I20260925 10:30:37.588572 1410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:37.616469 1410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:37.621544 1410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:37.621843 1410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:37.643510 1410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37155
I20260925 10:30:37.643517 1476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37155 every 8 connection(s)
I20260925 10:30:37.645160 1410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:37.649365 1477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:37.651383 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1410
I20260925 10:30:37.651594 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:37.652225 13281 external_mini_cluster.cc:1428] Setting key ea77164f1137a43a422736acb6a562a0
I20260925 10:30:37.661914 1477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224: Bootstrap starting.
I20260925 10:30:37.665870 1477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:37.667006 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:37.654182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60379 (local address 127.0.0.1:37155)
0925 10:30:37.655118 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:37.655131 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:37.655178 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:30:37.656993 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:37.657017 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:37.657429 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:37.657875 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:37.658507 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:37.660076 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:37.662160 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:37.662886 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:37.663031 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:37.664426 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:37.664434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:37.664449 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:37.665028 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:37.665106 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:37.665132 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:37.665262 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:37.665665 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 10:30:37.665883 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":458,"threads_started":1}
I20260925 10:30:37.667032 1477 log.cc:824] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:37.670287 1477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224: No bootstrap required, opened a new log
I20260925 10:30:37.671766 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:37.680676 1477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260925 10:30:37.681061 1477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:37.681213 1477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48084dc10b434b559d2480c82b635224, State: Initialized, Role: FOLLOWER
I20260925 10:30:37.681842 1477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260925 10:30:37.682128 1477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:37.682283 1477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:37.682471 1477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:37.684823 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:37.677399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56899 (local address 127.0.0.1:37155)
0925 10:30:37.677631 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:37.677638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:37.678242 (+ 604us) server_negotiation.cc:407] Connection header received
0925 10:30:37.678380 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:37.678386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:37.678466 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:37.678584 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:37.679120 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:37.680236 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:37.682044 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:37.682756 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:37.682896 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:37.684052 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:37.684058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:37.684062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:37.684088 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:37.684150 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:37.684171 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:37.684279 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:37.684458 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 10:30:37.684509 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 10:30:37.685351 1477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260925 10:30:37.686077 1477 leader_election.cc:302] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48084dc10b434b559d2480c82b635224; no voters:
I20260925 10:30:37.686709 1477 leader_election.cc:288] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:37.686861 1487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:37.687558 1487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [term 1 LEADER]: Becoming Leader. State: Replica: 48084dc10b434b559d2480c82b635224, State: Running, Role: LEADER
I20260925 10:30:37.688110 1487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260925 10:30:37.688843 1477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:37.691368 1488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48084dc10b434b559d2480c82b635224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } } }
I20260925 10:30:37.691787 1488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:37.692281 1489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48084dc10b434b559d2480c82b635224. Latest consensus state: current_term: 1 leader_uuid: "48084dc10b434b559d2480c82b635224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48084dc10b434b559d2480c82b635224" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } } }
I20260925 10:30:37.692587 1489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:37.693773 1493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:37.699246 1493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:37.708405 1493 catalog_manager.cc:1347] Generated new cluster ID: 321a2aaccfec44f3b09ba9221b270fbe
I20260925 10:30:37.708534 1493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:37.726392 1493 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:37.727849 1493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:37.741952 1493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48084dc10b434b559d2480c82b635224: Generated new TSK 0
I20260925 10:30:37.743043 1493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:37.750131 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1410
W20260925 10:30:37.751250 1472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60379
W20260925 10:30:37.751999 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1410: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:37.752199 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1410
W20260925 10:30:37.769722 1395 connection.cc:535] client connection to 127.0.0.1:37155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37155): Connection reset by peer (error 104)
2026-09-25T10:30:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:37 dist-test-slave-f9rl krb5kdc[1508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:37 dist-test-slave-f9rl krb5kdc[1508](info): set up 2 sockets
Sep 25 10:30:37 dist-test-slave-f9rl krb5kdc[1508](info): commencing operation
krb5kdc: starting...
W20260925 10:30:38.872455 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:39 dist-test-slave-f9rl krb5kdc[1508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:39.142493 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:39.487093 1521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:39.487450 1521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:39.487614 1521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:39.495756 1521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:39.495915 1521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:39.496025 1521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:39.496121 1521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:39.496191 1521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:39.506460 1521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:39.507834 1521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:39.509198 1521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:39.520953 1526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:39.521191 1527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:39.521545 1529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:39.522119 1521 server_base.cc:1034] running on GCE node
I20260925 10:30:39.522921 1521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:39.524849 1521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:39.526084 1521 hybrid_clock.cc:648] HybridClock initialized: now 1790332239526040 us; error 34 us; skew 500 ppm
I20260925 10:30:39.526499 1521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:39 dist-test-slave-f9rl krb5kdc[1508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:39.529845 1521 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:39.531358 1521 webserver.cc:460] Webserver started at http://127.0.0.1:44285/ using document root <none> and password file <none>
I20260925 10:30:39.532043 1521 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:39.532198 1521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:39.532521 1521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:39.535612 1521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "da8bce58647c4060afee28bb3c50658c"
format_stamp: "Formatted at 2026-09-25 10:30:39 on dist-test-slave-f9rl"
server_key: "af22ef1db491f674f155cb196cefa262"
server_key_iv: "dfec028c01dd3795419c1927959b39c4"
server_key_version: "encryptionkey@0"
I20260925 10:30:39.536434 1521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "da8bce58647c4060afee28bb3c50658c"
format_stamp: "Formatted at 2026-09-25 10:30:39 on dist-test-slave-f9rl"
server_key: "af22ef1db491f674f155cb196cefa262"
server_key_iv: "dfec028c01dd3795419c1927959b39c4"
server_key_version: "encryptionkey@0"
I20260925 10:30:39.541725 1521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 10:30:39.545259 1536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:39.546695 1521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:39.546895 1521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "da8bce58647c4060afee28bb3c50658c"
format_stamp: "Formatted at 2026-09-25 10:30:39 on dist-test-slave-f9rl"
server_key: "af22ef1db491f674f155cb196cefa262"
server_key_iv: "dfec028c01dd3795419c1927959b39c4"
server_key_version: "encryptionkey@0"
I20260925 10:30:39.547101 1521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:39.553382 1521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:39.557490 1521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:39.557799 1521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:39.577121 1521 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42275
I20260925 10:30:39.577175 1587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42275 every 8 connection(s)
I20260925 10:30:39.578601 1521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:39.582383 1588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:39.587945 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1521
I20260925 10:30:39.588160 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:39.588635 13281 external_mini_cluster.cc:1428] Setting key 8508c5379ebbdc5edb7fe13346c58848
I20260925 10:30:39.593410 1588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c: Bootstrap starting.
I20260925 10:30:39.596444 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:39.590447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48537 (local address 127.0.0.1:42275)
0925 10:30:39.591324 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:39.591336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:39.591376 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:30:39.593057 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:39.593078 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:39.593418 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:39.593845 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:39.594911 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:39.594916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:39.594929 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:39.594942 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:39.594995 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:39.595013 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:39.595189 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:39.595348 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:30:39.595555 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":407,"threads_started":1}
I20260925 10:30:39.597126 1588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:39.598309 1588 log.cc:824] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:39.600159 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:39.601256 1588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c: No bootstrap required, opened a new log
I20260925 10:30:39.607409 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:39.604799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41377 (local address 127.0.0.1:42275)
0925 10:30:39.604975 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:39.604981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:39.605474 (+ 493us) server_negotiation.cc:407] Connection header received
0925 10:30:39.605650 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:39.605655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:39.605721 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:39.605812 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:39.606750 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:39.606754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:39.606757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:39.606772 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:39.606812 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:39.606826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:39.606927 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:39.607049 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:30:39.607097 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:30:39.609817 1588 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260925 10:30:39.610200 1588 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:39.610306 1588 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8bce58647c4060afee28bb3c50658c, State: Initialized, Role: FOLLOWER
I20260925 10:30:39.610852 1588 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260925 10:30:39.611111 1588 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:39.611254 1588 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:39.611433 1588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:39.613670 1588 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260925 10:30:39.614167 1588 leader_election.cc:302] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8bce58647c4060afee28bb3c50658c; no voters:
I20260925 10:30:39.614785 1588 leader_election.cc:288] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:39.614916 1598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:39.615724 1598 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [term 1 LEADER]: Becoming Leader. State: Replica: da8bce58647c4060afee28bb3c50658c, State: Running, Role: LEADER
I20260925 10:30:39.616348 1598 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } }
I20260925 10:30:39.617046 1588 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:39.619570 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8bce58647c4060afee28bb3c50658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260925 10:30:39.619912 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:39.620718 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8bce58647c4060afee28bb3c50658c. Latest consensus state: current_term: 1 leader_uuid: "da8bce58647c4060afee28bb3c50658c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bce58647c4060afee28bb3c50658c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42275 } } }
I20260925 10:30:39.621076 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:39.622809 1605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:39.628144 1605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:39.637336 1605 catalog_manager.cc:1347] Generated new cluster ID: c41c4d5b1d7d4eed92973622265ffd52
I20260925 10:30:39.637470 1605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:39.653836 1605 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:39.654289 1605 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:39.670832 1605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8bce58647c4060afee28bb3c50658c: Generated new TSK 0
I20260925 10:30:39.671710 1605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:39.687678 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1521
W20260925 10:30:39.689208 1583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48537
W20260925 10:30:39.690047 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1521: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:39.690243 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1521
2026-09-25T10:30:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:39 dist-test-slave-f9rl krb5kdc[1619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:39 dist-test-slave-f9rl krb5kdc[1619](info): set up 2 sockets
Sep 25 10:30:39 dist-test-slave-f9rl krb5kdc[1619](info): commencing operation
krb5kdc: starting...
W20260925 10:30:40.807322 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:40 dist-test-slave-f9rl krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:41.073873 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:41.426927 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:41.427244 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:41.427337 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:41.434823 1632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:41.434943 1632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:41.434989 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:41.435055 1632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:41.435101 1632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:41.445128 1632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:41.446527 1632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:41.447926 1632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:41.460204 1637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:41.460316 1640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:41.460316 1638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:41.461547 1632 server_base.cc:1034] running on GCE node
I20260925 10:30:41.462376 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:41.463857 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:41.465142 1632 hybrid_clock.cc:648] HybridClock initialized: now 1790332241465092 us; error 32 us; skew 500 ppm
I20260925 10:30:41.465644 1632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:41 dist-test-slave-f9rl krb5kdc[1619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:41.469038 1632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:41.470861 1632 webserver.cc:460] Webserver started at http://127.0.0.1:36647/ using document root <none> and password file <none>
I20260925 10:30:41.471589 1632 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:41.471743 1632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:41.472054 1632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:41.475286 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "4975afba55934cb0b4fdb875ee2dcc3d"
format_stamp: "Formatted at 2026-09-25 10:30:41 on dist-test-slave-f9rl"
server_key: "cb9bed8400e8d5a97ac8441418a15c2f"
server_key_iv: "81b602cc8f0bc2b3b0220d2cd45eeb4e"
server_key_version: "encryptionkey@0"
I20260925 10:30:41.476090 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "4975afba55934cb0b4fdb875ee2dcc3d"
format_stamp: "Formatted at 2026-09-25 10:30:41 on dist-test-slave-f9rl"
server_key: "cb9bed8400e8d5a97ac8441418a15c2f"
server_key_iv: "81b602cc8f0bc2b3b0220d2cd45eeb4e"
server_key_version: "encryptionkey@0"
I20260925 10:30:41.481788 1632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:30:41.485373 1647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:41.486855 1632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:41.487053 1632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "4975afba55934cb0b4fdb875ee2dcc3d"
format_stamp: "Formatted at 2026-09-25 10:30:41 on dist-test-slave-f9rl"
server_key: "cb9bed8400e8d5a97ac8441418a15c2f"
server_key_iv: "81b602cc8f0bc2b3b0220d2cd45eeb4e"
server_key_version: "encryptionkey@0"
I20260925 10:30:41.487269 1632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:41.500653 1632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:41.505245 1632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:41.505613 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:41.523837 1632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33781
I20260925 10:30:41.523856 1698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33781 every 8 connection(s)
I20260925 10:30:41.525334 1632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:41.529336 1699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:41.535548 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1632
I20260925 10:30:41.535815 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:41.536276 13281 external_mini_cluster.cc:1428] Setting key e1b1c7ae2ac2ff8350e26e3e328b7605
I20260925 10:30:41.540761 1699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d: Bootstrap starting.
I20260925 10:30:41.544641 1699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:41.545920 1699 log.cc:824] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:41.549098 1699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d: No bootstrap required, opened a new log
Sep 25 10:30:41 dist-test-slave-f9rl krb5kdc[1619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:30:41.560556 1699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260925 10:30:41.561100 1699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:41.561337 1699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4975afba55934cb0b4fdb875ee2dcc3d, State: Initialized, Role: FOLLOWER
I20260925 10:30:41.562240 1699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260925 10:30:41.562559 1699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:41.562703 1699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:41.562901 1699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:41.564061 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:41.538256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49883 (local address 127.0.0.1:33781)
0925 10:30:41.539073 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:41.539090 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:41.539139 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:30:41.541572 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:41.541600 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:41.542102 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:41.542551 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:41.543888 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:41.545918 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:41.547404 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:41.548435 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:41.548623 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:41.553993 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:41.554005 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:41.554027 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:41.554085 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:41.558568 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:41.559443 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:41.559452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:41.559463 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:41.559711 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:41.560154 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:41.560159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:41.560162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:41.560972 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:41.561307 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:41.561965 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 10:30:41.562470 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":350,"threads_started":1}
I20260925 10:30:41.566535 1699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260925 10:30:41.567173 1699 leader_election.cc:302] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4975afba55934cb0b4fdb875ee2dcc3d; no voters:
I20260925 10:30:41.567883 1699 leader_election.cc:288] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:41.568012 1704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:41.568809 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:41.569489 1704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [term 1 LEADER]: Becoming Leader. State: Replica: 4975afba55934cb0b4fdb875ee2dcc3d, State: Running, Role: LEADER
I20260925 10:30:41.570076 1704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260925 10:30:41.570780 1699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:41.573287 1706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4975afba55934cb0b4fdb875ee2dcc3d. Latest consensus state: current_term: 1 leader_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } } }
I20260925 10:30:41.573911 1706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:41.574600 1705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4975afba55934cb0b4fdb875ee2dcc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } } }
I20260925 10:30:41.574942 1705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:41.575001 1715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:41.593719 1715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:41.608956 1715 catalog_manager.cc:1347] Generated new cluster ID: 9d4dfc7d47d14c30b29d2f3d52c51bb4
I20260925 10:30:41.609208 1715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:41.610930 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:41.591807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36649 (local address 127.0.0.1:33781)
0925 10:30:41.592142 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:41.592152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:41.594059 (+ 1907us) server_negotiation.cc:407] Connection header received
0925 10:30:41.594285 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:41.594291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:41.594376 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:41.594488 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:41.596052 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:41.596817 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:41.598674 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:41.599426 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:41.599533 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:41.603094 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:41.603101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:41.603106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:41.603147 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:41.607076 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:41.608010 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:41.608015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:41.608018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:41.608172 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:41.608680 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:41.608685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:41.608688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:41.609092 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:41.609333 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:41.610083 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 10:30:41.610327 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260925 10:30:41.620450 1715 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:41.621799 1715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:41.636778 1715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4975afba55934cb0b4fdb875ee2dcc3d: Generated new TSK 0
I20260925 10:30:41.637856 1715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:41.648052 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1632
I20260925 10:30:41.649803 1694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:41.740594 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1632
2026-09-25T10:30:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:41 dist-test-slave-f9rl krb5kdc[1731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:41 dist-test-slave-f9rl krb5kdc[1731](info): set up 2 sockets
Sep 25 10:30:41 dist-test-slave-f9rl krb5kdc[1731](info): commencing operation
krb5kdc: starting...
W20260925 10:30:42.859341 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:43 dist-test-slave-f9rl krb5kdc[1731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:43.124392 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:43.457193 1744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:43.457561 1744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:43.457674 1744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:43.465389 1744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:43.465574 1744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:43.465636 1744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:43.465699 1744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:43.465744 1744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:43.475664 1744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:43.477033 1744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:43.478407 1744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:43.490721 1749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:43.490675 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:43.490635 1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:43.491593 1744 server_base.cc:1034] running on GCE node
I20260925 10:30:43.492406 1744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:43.494033 1744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:43.495296 1744 hybrid_clock.cc:648] HybridClock initialized: now 1790332243495242 us; error 50 us; skew 500 ppm
I20260925 10:30:43.495680 1744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:43 dist-test-slave-f9rl krb5kdc[1731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:43.503965 1744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:43.505569 1744 webserver.cc:460] Webserver started at http://127.0.0.1:38695/ using document root <none> and password file <none>
I20260925 10:30:43.506286 1744 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:43.506445 1744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:43.506767 1744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:43.509893 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "40b0b461e3434c509d81aa8ed1d4b61e"
format_stamp: "Formatted at 2026-09-25 10:30:43 on dist-test-slave-f9rl"
server_key: "415fcaca1dd4bd617883def54fdb6d3d"
server_key_iv: "19d15bcb4b52151751d34413800b55e3"
server_key_version: "encryptionkey@0"
I20260925 10:30:43.510679 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "40b0b461e3434c509d81aa8ed1d4b61e"
format_stamp: "Formatted at 2026-09-25 10:30:43 on dist-test-slave-f9rl"
server_key: "415fcaca1dd4bd617883def54fdb6d3d"
server_key_iv: "19d15bcb4b52151751d34413800b55e3"
server_key_version: "encryptionkey@0"
I20260925 10:30:43.516403 1744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:43.520044 1759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:43.521389 1744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 10:30:43.521639 1744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "40b0b461e3434c509d81aa8ed1d4b61e"
format_stamp: "Formatted at 2026-09-25 10:30:43 on dist-test-slave-f9rl"
server_key: "415fcaca1dd4bd617883def54fdb6d3d"
server_key_iv: "19d15bcb4b52151751d34413800b55e3"
server_key_version: "encryptionkey@0"
I20260925 10:30:43.521862 1744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:43.571477 1744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:43.575995 1744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:43.576311 1744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:43.595158 1744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39525
I20260925 10:30:43.595191 1810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39525 every 8 connection(s)
I20260925 10:30:43.596554 1744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:43.599756 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1744
I20260925 10:30:43.599972 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:43.600611 13281 external_mini_cluster.cc:1428] Setting key 6b75e0e037fe974b52a9f4df65f14717
I20260925 10:30:43.601282 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:43.615839 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:43.602707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42559 (local address 127.0.0.1:39525)
0925 10:30:43.603747 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:43.603763 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:43.604160 (+ 397us) server_negotiation.cc:407] Connection header received
0925 10:30:43.606569 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:43.606590 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:43.607027 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:43.607356 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:43.608023 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:43.609354 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:43.611284 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:43.611958 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:43.612090 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:43.613388 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:43.613396 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:43.613410 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:43.614033 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:43.614114 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:43.614141 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:43.614227 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:43.614458 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:30:43.614766 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":421,"threads_started":1}
I20260925 10:30:43.616349 1811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e: Bootstrap starting.
I20260925 10:30:43.619899 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:43.620069 1811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:43.621264 1811 log.cc:824] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:43.624807 1811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e: No bootstrap required, opened a new log
I20260925 10:30:43.632143 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:43.625364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41113 (local address 127.0.0.1:39525)
0925 10:30:43.625618 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:43.625626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:43.625990 (+ 364us) server_negotiation.cc:407] Connection header received
0925 10:30:43.626236 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:43.626242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:43.626336 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:43.626450 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:43.627074 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:43.628229 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:43.629521 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:43.630276 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:43.630421 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:43.631387 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:43.631394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:43.631398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:43.631425 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:43.631472 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:43.631502 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:43.631630 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:43.631688 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:30:43.631782 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 10:30:43.634931 1811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260925 10:30:43.635298 1811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:43.635372 1811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b0b461e3434c509d81aa8ed1d4b61e, State: Initialized, Role: FOLLOWER
I20260925 10:30:43.635861 1811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260925 10:30:43.636113 1811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:43.636216 1811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:43.636340 1811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:43.638945 1811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260925 10:30:43.639392 1811 leader_election.cc:302] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b0b461e3434c509d81aa8ed1d4b61e; no voters:
I20260925 10:30:43.639909 1811 leader_election.cc:288] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:43.640071 1821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:43.640750 1821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [term 1 LEADER]: Becoming Leader. State: Replica: 40b0b461e3434c509d81aa8ed1d4b61e, State: Running, Role: LEADER
I20260925 10:30:43.641345 1821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260925 10:30:43.642237 1811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:43.644134 1822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260925 10:30:43.644603 1822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:43.644119 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b0b461e3434c509d81aa8ed1d4b61e. Latest consensus state: current_term: 1 leader_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b0b461e3434c509d81aa8ed1d4b61e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260925 10:30:43.650142 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:43.653120 1826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:43.658845 1826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:43.668296 1826 catalog_manager.cc:1347] Generated new cluster ID: deb0851d3c734902a814017c66b4192e
I20260925 10:30:43.668429 1826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:43.700327 1826 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:43.701754 1826 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:43.707516 1826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b0b461e3434c509d81aa8ed1d4b61e: Generated new TSK 0
I20260925 10:30:43.708352 1826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:43.723341 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1744
W20260925 10:30:43.724498 1806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42559
W20260925 10:30:43.725238 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1744: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:43.725402 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1744
2026-09-25T10:30:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:43 dist-test-slave-f9rl krb5kdc[1842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:43 dist-test-slave-f9rl krb5kdc[1842](info): set up 2 sockets
Sep 25 10:30:43 dist-test-slave-f9rl krb5kdc[1842](info): commencing operation
krb5kdc: starting...
W20260925 10:30:44.831378 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:45 dist-test-slave-f9rl krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:45.093603 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:45.431787 1855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:45.432097 1855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:45.432271 1855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:45.439669 1855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:45.439828 1855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:45.439922 1855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:45.440034 1855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:45.440120 1855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:45.450367 1855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:45.451802 1855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:45.453173 1855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:45.465399 1860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:45.465498 1861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:45.465345 1863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:45.471024 1855 server_base.cc:1034] running on GCE node
I20260925 10:30:45.471895 1855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:45.473472 1855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:45.474747 1855 hybrid_clock.cc:648] HybridClock initialized: now 1790332245474697 us; error 39 us; skew 500 ppm
I20260925 10:30:45.475134 1855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:45 dist-test-slave-f9rl krb5kdc[1842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:45.478302 1855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:45.479868 1855 webserver.cc:460] Webserver started at http://127.0.0.1:38835/ using document root <none> and password file <none>
I20260925 10:30:45.480542 1855 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:45.480648 1855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:45.480934 1855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:45.483985 1855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "8c11126d17584283830c437bfc65641f"
format_stamp: "Formatted at 2026-09-25 10:30:45 on dist-test-slave-f9rl"
server_key: "52ec5d1b43d6a94ee93805ab34b315e2"
server_key_iv: "97005c4a03cce85bdf03904b0cba5916"
server_key_version: "encryptionkey@0"
I20260925 10:30:45.484737 1855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "8c11126d17584283830c437bfc65641f"
format_stamp: "Formatted at 2026-09-25 10:30:45 on dist-test-slave-f9rl"
server_key: "52ec5d1b43d6a94ee93805ab34b315e2"
server_key_iv: "97005c4a03cce85bdf03904b0cba5916"
server_key_version: "encryptionkey@0"
I20260925 10:30:45.490124 1855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 10:30:45.493750 1870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:45.495332 1855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:45.495545 1855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "8c11126d17584283830c437bfc65641f"
format_stamp: "Formatted at 2026-09-25 10:30:45 on dist-test-slave-f9rl"
server_key: "52ec5d1b43d6a94ee93805ab34b315e2"
server_key_iv: "97005c4a03cce85bdf03904b0cba5916"
server_key_version: "encryptionkey@0"
I20260925 10:30:45.495774 1855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:45.501732 1855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:45.505812 1855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:45.506125 1855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:45.525264 1855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41305
I20260925 10:30:45.525321 1922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41305 every 8 connection(s)
I20260925 10:30:45.526729 1855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:45.530550 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:45.532437 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1855
I20260925 10:30:45.532625 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:45.533165 13281 external_mini_cluster.cc:1428] Setting key 78c6773169fc8364c3122f811e993fc8
I20260925 10:30:45.541262 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:45.535270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39531 (local address 127.0.0.1:41305)
0925 10:30:45.536211 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:45.536226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:45.536273 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:30:45.537922 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:45.537949 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:45.538298 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:45.538624 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:45.539680 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:45.539686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:45.539698 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:45.539759 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:45.539826 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:45.539845 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:45.540029 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:45.540130 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:30:45.540327 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":509,"threads_started":1}
I20260925 10:30:45.544497 1923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f: Bootstrap starting.
I20260925 10:30:45.544971 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:45.548439 1923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:45.549922 1923 log.cc:824] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:45.552712 1923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f: No bootstrap required, opened a new log
I20260925 10:30:45.553018 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:45.550235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34069 (local address 127.0.0.1:41305)
0925 10:30:45.550425 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:45.550434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:45.550982 (+ 548us) server_negotiation.cc:407] Connection header received
0925 10:30:45.551182 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:45.551189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:45.551248 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:45.551353 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:45.552291 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:45.552297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:45.552302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:45.552319 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:45.552367 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:45.552382 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:45.552500 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:45.552623 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:30:45.552685 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:30:45.561156 1923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } }
I20260925 10:30:45.561643 1923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:45.561769 1923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c11126d17584283830c437bfc65641f, State: Initialized, Role: FOLLOWER
I20260925 10:30:45.562381 1923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } }
I20260925 10:30:45.562610 1923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:45.562770 1923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:45.562955 1923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:45.565349 1923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } }
I20260925 10:30:45.565948 1923 leader_election.cc:302] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c11126d17584283830c437bfc65641f; no voters:
I20260925 10:30:45.566707 1923 leader_election.cc:288] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:45.566835 1933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:45.567565 1933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [term 1 LEADER]: Becoming Leader. State: Replica: 8c11126d17584283830c437bfc65641f, State: Running, Role: LEADER
I20260925 10:30:45.568135 1933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } }
I20260925 10:30:45.568964 1923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:45.571310 1934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c11126d17584283830c437bfc65641f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } } }
I20260925 10:30:45.571808 1934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:45.572139 1935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c11126d17584283830c437bfc65641f. Latest consensus state: current_term: 1 leader_uuid: "8c11126d17584283830c437bfc65641f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c11126d17584283830c437bfc65641f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41305 } } }
I20260925 10:30:45.572500 1935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:45.574518 1941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:45.580355 1941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:45.589125 1941 catalog_manager.cc:1347] Generated new cluster ID: 264d383718c742ed88d02e4b65504f12
I20260925 10:30:45.589259 1941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:45.623400 1941 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:45.623957 1941 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:45.642984 1941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c11126d17584283830c437bfc65641f: Generated new TSK 0
I20260925 10:30:45.643940 1941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:45.661867 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1855
W20260925 10:30:45.663327 1918 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39531
W20260925 10:30:45.664134 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1855: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:45.664259 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1855
2026-09-25T10:30:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:45 dist-test-slave-f9rl krb5kdc[1954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:45 dist-test-slave-f9rl krb5kdc[1954](info): set up 2 sockets
Sep 25 10:30:45 dist-test-slave-f9rl krb5kdc[1954](info): commencing operation
krb5kdc: starting...
W20260925 10:30:46.767198 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:46 dist-test-slave-f9rl krb5kdc[1954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:47.030783 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:47.368394 1967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:47.368695 1967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:47.368883 1967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:47.376152 1967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:47.376274 1967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:47.376394 1967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:47.376506 1967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:47.376588 1967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:47.386739 1967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:47.388142 1967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:47.389536 1967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:47.401863 1972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:47.401917 1975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:47.401955 1973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:47.402763 1967 server_base.cc:1034] running on GCE node
I20260925 10:30:47.403513 1967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:47.404968 1967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:47.406190 1967 hybrid_clock.cc:648] HybridClock initialized: now 1790332247406139 us; error 37 us; skew 500 ppm
I20260925 10:30:47.406579 1967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:47 dist-test-slave-f9rl krb5kdc[1954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:47.409883 1967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:47.411444 1967 webserver.cc:460] Webserver started at http://127.0.0.1:43479/ using document root <none> and password file <none>
I20260925 10:30:47.412169 1967 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:47.412322 1967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:47.412633 1967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:47.415697 1967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "2cac98e23d9e4ddabce66ab08d96d8b8"
format_stamp: "Formatted at 2026-09-25 10:30:47 on dist-test-slave-f9rl"
server_key: "7fdc9ac6e48d4293c7fa48a95f7b4b23"
server_key_iv: "74861a1be56b77760f18839e7d5e8bc4"
server_key_version: "encryptionkey@0"
I20260925 10:30:47.416499 1967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "2cac98e23d9e4ddabce66ab08d96d8b8"
format_stamp: "Formatted at 2026-09-25 10:30:47 on dist-test-slave-f9rl"
server_key: "7fdc9ac6e48d4293c7fa48a95f7b4b23"
server_key_iv: "74861a1be56b77760f18839e7d5e8bc4"
server_key_version: "encryptionkey@0"
I20260925 10:30:47.421723 1967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:30:47.425179 1982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:47.426586 1967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:47.426797 1967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "2cac98e23d9e4ddabce66ab08d96d8b8"
format_stamp: "Formatted at 2026-09-25 10:30:47 on dist-test-slave-f9rl"
server_key: "7fdc9ac6e48d4293c7fa48a95f7b4b23"
server_key_iv: "74861a1be56b77760f18839e7d5e8bc4"
server_key_version: "encryptionkey@0"
I20260925 10:30:47.427006 1967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:47.444793 1967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:47.449409 1967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:47.449811 1967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:47.469378 1967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39953
I20260925 10:30:47.469417 2033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39953 every 8 connection(s)
I20260925 10:30:47.470899 1967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:47.474861 2034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:47.475921 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 1967
I20260925 10:30:47.476125 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:47.476706 13281 external_mini_cluster.cc:1428] Setting key 55f6b0eccea768b9edd0628375516109
I20260925 10:30:47.489787 2034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8: Bootstrap starting.
Sep 25 10:30:47 dist-test-slave-f9rl krb5kdc[1954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:30:47.493821 2034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:47.495272 2034 log.cc:824] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:47.498870 2034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8: No bootstrap required, opened a new log
I20260925 10:30:47.502007 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:47.478714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:43519 (local address 127.0.0.1:39953)
0925 10:30:47.479515 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:47.479527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:47.479578 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:30:47.481363 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:47.481389 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:47.481898 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:47.482242 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:47.484033 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:47.485339 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:47.487204 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:47.487908 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:47.488046 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:47.493526 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:47.493541 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:47.493567 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:47.493623 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:47.497520 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:47.498390 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:47.498406 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:47.498412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:47.498579 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:47.499112 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:47.499116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:47.499118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:47.499709 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:47.500053 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:47.500565 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 10:30:47.500890 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260925 10:30:47.506080 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:47.507413 2034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260925 10:30:47.507802 2034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:47.507910 2034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cac98e23d9e4ddabce66ab08d96d8b8, State: Initialized, Role: FOLLOWER
I20260925 10:30:47.508464 2034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260925 10:30:47.508767 2034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:47.508932 2034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:47.509123 2034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:47.512085 2034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260925 10:30:47.512604 2034 leader_election.cc:302] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cac98e23d9e4ddabce66ab08d96d8b8; no voters:
I20260925 10:30:47.513242 2034 leader_election.cc:288] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:47.513407 2044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:47.514267 2044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2cac98e23d9e4ddabce66ab08d96d8b8, State: Running, Role: LEADER
I20260925 10:30:47.514921 2044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } }
I20260925 10:30:47.516922 2044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cac98e23d9e4ddabce66ab08d96d8b8. Latest consensus state: current_term: 1 leader_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } } }
I20260925 10:30:47.518527 2044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:47.518172 2046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cac98e23d9e4ddabce66ab08d96d8b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39953 } } }
I20260925 10:30:47.518803 2046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:47.521507 2034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:47.526034 2048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:47.531499 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:47.510746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:41271 (local address 127.0.0.1:39953)
0925 10:30:47.510938 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:47.510944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:47.511343 (+ 399us) server_negotiation.cc:407] Connection header received
0925 10:30:47.511479 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:47.511484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:47.511562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:47.511670 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:47.513499 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:47.515181 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:47.517155 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:47.517970 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:47.518124 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:47.522004 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:47.522012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:47.522018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:47.522070 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:47.526835 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:47.528014 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:47.528023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:47.528029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:47.528207 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:47.528812 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:47.528817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:47.528820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:47.529244 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:47.530735 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:47.530909 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:30:47.531108 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:30:47.531772 2048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:47.542785 2048 catalog_manager.cc:1347] Generated new cluster ID: 298438810d304cb9b20e8323ffc95ebf
I20260925 10:30:47.542954 2048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:47.551964 2048 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:47.553234 2048 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:47.563289 2048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cac98e23d9e4ddabce66ab08d96d8b8: Generated new TSK 0
I20260925 10:30:47.564190 2048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:47.580256 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 1967
I20260925 10:30:47.581418 2029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:47.673539 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 1967
W20260925 10:30:47.691375 1952 connection.cc:535] client connection to 127.0.0.1:39953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39953): Connection reset by peer (error 104)
2026-09-25T10:30:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2010 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:47 dist-test-slave-f9rl krb5kdc[2066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:47 dist-test-slave-f9rl krb5kdc[2066](info): set up 2 sockets
Sep 25 10:30:47 dist-test-slave-f9rl krb5kdc[2066](info): commencing operation
krb5kdc: starting...
W20260925 10:30:48.823324 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:48 dist-test-slave-f9rl krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:49.090401 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:49.438184 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:49.438486 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:49.438642 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:49.446007 2079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:49.446141 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:49.446236 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:49.446346 2079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:49.446422 2079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:49.456465 2079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:49.457921 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:49.459286 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:49.471778 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:49.471863 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:49.471840 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:49.472935 2079 server_base.cc:1034] running on GCE node
I20260925 10:30:49.474032 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:49.475782 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:49.477082 2079 hybrid_clock.cc:648] HybridClock initialized: now 1790332249477022 us; error 45 us; skew 500 ppm
I20260925 10:30:49.477494 2079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:49 dist-test-slave-f9rl krb5kdc[2066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:49.488524 2079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:49.490597 2079 webserver.cc:460] Webserver started at http://127.0.0.1:40167/ using document root <none> and password file <none>
I20260925 10:30:49.491400 2079 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:49.491562 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:49.491905 2079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:49.495426 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "951d5005c15f4d1eb663e90dfe698129"
format_stamp: "Formatted at 2026-09-25 10:30:49 on dist-test-slave-f9rl"
server_key: "66485d073cd7fb712482bccf54d09ac4"
server_key_iv: "591e12efe0e1293e52cb8fda97c1a726"
server_key_version: "encryptionkey@0"
I20260925 10:30:49.496315 2079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "951d5005c15f4d1eb663e90dfe698129"
format_stamp: "Formatted at 2026-09-25 10:30:49 on dist-test-slave-f9rl"
server_key: "66485d073cd7fb712482bccf54d09ac4"
server_key_iv: "591e12efe0e1293e52cb8fda97c1a726"
server_key_version: "encryptionkey@0"
I20260925 10:30:49.502267 2079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260925 10:30:49.506209 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:49.507753 2079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:49.507948 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "951d5005c15f4d1eb663e90dfe698129"
format_stamp: "Formatted at 2026-09-25 10:30:49 on dist-test-slave-f9rl"
server_key: "66485d073cd7fb712482bccf54d09ac4"
server_key_iv: "591e12efe0e1293e52cb8fda97c1a726"
server_key_version: "encryptionkey@0"
I20260925 10:30:49.508164 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:49.560454 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:49.565033 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:49.565362 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:49.585901 2079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36845
I20260925 10:30:49.585939 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36845 every 8 connection(s)
I20260925 10:30:49.587471 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:49.590189 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2079
I20260925 10:30:49.590399 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:49.590958 13281 external_mini_cluster.cc:1428] Setting key 4c62772d16fdd15b0ea896e57efab0ee
I20260925 10:30:49.591969 2146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:49.605618 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:49.592946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:45681 (local address 127.0.0.1:36845)
0925 10:30:49.593885 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:49.593898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:49.593943 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:30:49.595650 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:49.595674 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:49.596130 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:49.596535 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:49.597145 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:49.598790 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:49.600651 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:49.601382 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:49.601683 (+ 301us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:49.602946 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:49.602954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:49.602972 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:49.603582 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:49.603674 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:49.603704 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:49.603831 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:49.604260 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 10:30:49.604472 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":474,"threads_started":1}
I20260925 10:30:49.606155 2146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129: Bootstrap starting.
I20260925 10:30:49.609817 2146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:49.609972 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:49.611054 2146 log.cc:824] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:49.614800 2146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129: No bootstrap required, opened a new log
I20260925 10:30:49.622773 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:49.615314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:56081 (local address 127.0.0.1:36845)
0925 10:30:49.615499 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:49.615505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:49.616113 (+ 608us) server_negotiation.cc:407] Connection header received
0925 10:30:49.616255 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:49.616262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:49.616348 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:49.616469 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:49.616980 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:49.618233 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:49.619507 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:49.620276 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:49.620422 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:49.621893 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:49.621900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:49.621905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:49.621927 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:49.621980 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:49.622005 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:49.622124 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:49.622371 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 10:30:49.622429 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:30:49.624642 2146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260925 10:30:49.624993 2146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:49.625066 2146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951d5005c15f4d1eb663e90dfe698129, State: Initialized, Role: FOLLOWER
I20260925 10:30:49.625679 2146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260925 10:30:49.626021 2146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:49.626186 2146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:49.626406 2146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:49.628698 2146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260925 10:30:49.629153 2146 leader_election.cc:302] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951d5005c15f4d1eb663e90dfe698129; no voters:
I20260925 10:30:49.629765 2146 leader_election.cc:288] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:49.629906 2156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:49.630669 2156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [term 1 LEADER]: Becoming Leader. State: Replica: 951d5005c15f4d1eb663e90dfe698129, State: Running, Role: LEADER
I20260925 10:30:49.631230 2156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260925 10:30:49.631938 2146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:49.634312 2157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951d5005c15f4d1eb663e90dfe698129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260925 10:30:49.634733 2157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:49.635713 2161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:49.634334 2158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951d5005c15f4d1eb663e90dfe698129. Latest consensus state: current_term: 1 leader_uuid: "951d5005c15f4d1eb663e90dfe698129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951d5005c15f4d1eb663e90dfe698129" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260925 10:30:49.636941 2158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:49.642297 2161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:49.653648 2161 catalog_manager.cc:1347] Generated new cluster ID: 4a3b8634d8064f9f840923a748c24357
I20260925 10:30:49.653780 2161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:49.670168 2161 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:49.671480 2161 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:49.680903 2161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 951d5005c15f4d1eb663e90dfe698129: Generated new TSK 0
I20260925 10:30:49.681770 2161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:49.692179 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2079
W20260925 10:30:49.693156 2141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.116:45681
W20260925 10:30:49.693928 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2079: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:49.694062 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2079
W20260925 10:30:49.711333 2064 connection.cc:535] client connection to 127.0.0.1:36845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36845): Connection reset by peer (error 104)
2026-09-25T10:30:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2020 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:49 dist-test-slave-f9rl krb5kdc[2177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:49 dist-test-slave-f9rl krb5kdc[2177](info): set up 2 sockets
Sep 25 10:30:49 dist-test-slave-f9rl krb5kdc[2177](info): commencing operation
krb5kdc: starting...
W20260925 10:30:50.896060 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:51 dist-test-slave-f9rl krb5kdc[2177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:51.164935 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:51.500161 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:51.500468 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:51.500641 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:51.508034 2190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:51.508153 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:51.508234 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:51.508342 2190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:51.508427 2190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:51.518788 2190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:51.520133 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:51.521515 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:51.533555 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:51.533644 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:51.533631 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:51.534480 2190 server_base.cc:1034] running on GCE node
I20260925 10:30:51.535324 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:51.536723 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:51.537953 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790332251537911 us; error 34 us; skew 500 ppm
I20260925 10:30:51.538321 2190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:51 dist-test-slave-f9rl krb5kdc[2177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:51.541323 2190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:51.542760 2190 webserver.cc:460] Webserver started at http://127.0.0.1:42765/ using document root <none> and password file <none>
I20260925 10:30:51.543401 2190 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:51.543522 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:51.543782 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:51.546769 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "3131a42a310444289aae7fd82ed955f1"
format_stamp: "Formatted at 2026-09-25 10:30:51 on dist-test-slave-f9rl"
server_key: "69803af9a9654fdc13b27fe629ef5869"
server_key_iv: "81dd774bb9de3f87d1da5262e5a0abb3"
server_key_version: "encryptionkey@0"
I20260925 10:30:51.547487 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "3131a42a310444289aae7fd82ed955f1"
format_stamp: "Formatted at 2026-09-25 10:30:51 on dist-test-slave-f9rl"
server_key: "69803af9a9654fdc13b27fe629ef5869"
server_key_iv: "81dd774bb9de3f87d1da5262e5a0abb3"
server_key_version: "encryptionkey@0"
I20260925 10:30:51.552645 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:30:51.556013 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:51.557333 2190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:30:51.557536 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "3131a42a310444289aae7fd82ed955f1"
format_stamp: "Formatted at 2026-09-25 10:30:51 on dist-test-slave-f9rl"
server_key: "69803af9a9654fdc13b27fe629ef5869"
server_key_iv: "81dd774bb9de3f87d1da5262e5a0abb3"
server_key_version: "encryptionkey@0"
I20260925 10:30:51.557736 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:51.563621 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:51.567584 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:51.567889 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:51.586539 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37197
I20260925 10:30:51.586581 2256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37197 every 8 connection(s)
I20260925 10:30:51.587987 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:51.590304 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2190
I20260925 10:30:51.590492 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:51.590973 13281 external_mini_cluster.cc:1428] Setting key 43aa10d3834f65f6399855cc03c57243
I20260925 10:30:51.592087 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:51.599484 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:51.592708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:58481 (local address 127.0.0.1:37197)
0925 10:30:51.593554 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:51.593569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:51.593621 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:30:51.595550 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:51.595580 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:51.596044 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:51.596495 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:51.597538 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:51.597549 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:51.597573 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:51.597590 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:51.597674 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:51.597704 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:51.597916 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:51.598052 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:30:51.598253 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260925 10:30:51.603039 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:51.605610 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1: Bootstrap starting.
I20260925 10:30:51.609115 2257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:51.609936 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:51.607394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:50607 (local address 127.0.0.1:37197)
0925 10:30:51.607577 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:51.607583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:51.608059 (+ 476us) server_negotiation.cc:407] Connection header received
0925 10:30:51.608194 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:51.608199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:51.608266 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:51.608376 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:51.609214 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:51.609219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:51.609223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:51.609241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:51.609283 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:51.609339 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:51.609490 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:51.609594 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 10:30:51.609670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:30:51.610467 2257 log.cc:824] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:51.613129 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1: No bootstrap required, opened a new log
I20260925 10:30:51.620352 2257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260925 10:30:51.620765 2257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:51.620895 2257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3131a42a310444289aae7fd82ed955f1, State: Initialized, Role: FOLLOWER
I20260925 10:30:51.621510 2257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260925 10:30:51.621752 2257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:51.621902 2257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:51.622073 2257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:51.624238 2257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260925 10:30:51.624758 2257 leader_election.cc:302] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3131a42a310444289aae7fd82ed955f1; no voters:
I20260925 10:30:51.625324 2257 leader_election.cc:288] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:51.625476 2267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:51.626140 2267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3131a42a310444289aae7fd82ed955f1, State: Running, Role: LEADER
I20260925 10:30:51.626715 2267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } }
I20260925 10:30:51.627439 2257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:51.630147 2269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3131a42a310444289aae7fd82ed955f1. Latest consensus state: current_term: 1 leader_uuid: "3131a42a310444289aae7fd82ed955f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } } }
I20260925 10:30:51.630470 2269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:51.629464 2268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3131a42a310444289aae7fd82ed955f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3131a42a310444289aae7fd82ed955f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37197 } } }
I20260925 10:30:51.631012 2268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:51.634827 2275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:51.640224 2275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:51.649029 2275 catalog_manager.cc:1347] Generated new cluster ID: ed77805049e2449fa5887ff85515e887
I20260925 10:30:51.649154 2275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:51.679833 2275 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:51.680282 2275 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:51.687702 2275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3131a42a310444289aae7fd82ed955f1: Generated new TSK 0
I20260925 10:30:51.688807 2275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:51.696636 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2190
W20260925 10:30:51.697887 2252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.116:58481
W20260925 10:30:51.698633 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:51.698791 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2190
2026-09-25T10:30:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:51 dist-test-slave-f9rl krb5kdc[2288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:51 dist-test-slave-f9rl krb5kdc[2288](info): set up 2 sockets
Sep 25 10:30:51 dist-test-slave-f9rl krb5kdc[2288](info): commencing operation
krb5kdc: starting...
W20260925 10:30:52.891446 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:53.173285 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:53.520699 2301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:53.521023 2301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:53.521121 2301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:53.528811 2301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:53.528937 2301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:53.528988 2301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:53.529053 2301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:53.529098 2301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:53.540057 2301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:53.541356 2301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:53.542918 2301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:53.555397 2306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:53.555486 2309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:53.555516 2307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:53.556082 2301 server_base.cc:1034] running on GCE node
I20260925 10:30:53.556881 2301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:53.558385 2301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:53.559628 2301 hybrid_clock.cc:648] HybridClock initialized: now 1790332253559579 us; error 56 us; skew 500 ppm
I20260925 10:30:53.560009 2301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:53.563380 2301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:53.564981 2301 webserver.cc:460] Webserver started at http://127.0.0.1:38395/ using document root <none> and password file <none>
I20260925 10:30:53.565738 2301 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:53.565889 2301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:53.566195 2301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:53.569205 2301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "6e0a720644ed4dfd8b82b4fc0c507290"
format_stamp: "Formatted at 2026-09-25 10:30:53 on dist-test-slave-f9rl"
server_key: "b2bf5ee33c84ec66ce97a419d516cc64"
server_key_iv: "ed09db77c39076fe0f13e3530c94c5e2"
server_key_version: "encryptionkey@0"
I20260925 10:30:53.570065 2301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "6e0a720644ed4dfd8b82b4fc0c507290"
format_stamp: "Formatted at 2026-09-25 10:30:53 on dist-test-slave-f9rl"
server_key: "b2bf5ee33c84ec66ce97a419d516cc64"
server_key_iv: "ed09db77c39076fe0f13e3530c94c5e2"
server_key_version: "encryptionkey@0"
I20260925 10:30:53.575487 2301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:30:53.579097 2316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:53.580466 2301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 10:30:53.580687 2301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "6e0a720644ed4dfd8b82b4fc0c507290"
format_stamp: "Formatted at 2026-09-25 10:30:53 on dist-test-slave-f9rl"
server_key: "b2bf5ee33c84ec66ce97a419d516cc64"
server_key_iv: "ed09db77c39076fe0f13e3530c94c5e2"
server_key_version: "encryptionkey@0"
I20260925 10:30:53.580895 2301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:53.599898 2301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:53.604323 2301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:53.604629 2301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:53.624414 2301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45665
I20260925 10:30:53.624439 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45665 every 8 connection(s)
I20260925 10:30:53.625890 2301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:53.629618 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2301
I20260925 10:30:53.629812 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:53.630102 2368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:53.630396 13281 external_mini_cluster.cc:1428] Setting key 989574c916aec64ce4bd8e33ff3ce64e
I20260925 10:30:53.644865 2368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290: Bootstrap starting.
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:30:53.648859 2368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:53.650072 2368 log.cc:824] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:53.653985 2368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290: No bootstrap required, opened a new log
I20260925 10:30:53.654978 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:53.632198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:59491 (local address 127.0.0.1:45665)
0925 10:30:53.632959 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:53.632974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:53.633030 (+ 56us) server_negotiation.cc:407] Connection header received
0925 10:30:53.635326 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:53.635354 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:53.635907 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:53.636356 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:53.637775 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:53.639716 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:53.640961 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:53.642083 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:53.642281 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:53.646663 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:53.646671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:53.646685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:53.646733 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:53.649917 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:53.651023 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:53.651034 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:53.651045 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:53.651269 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:53.651972 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:53.651976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:53.651978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:53.652545 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:53.652806 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:53.653533 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 10:30:53.653902 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":298,"threads_started":1}
I20260925 10:30:53.658995 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:53.664984 2368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260925 10:30:53.665480 2368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:53.665642 2368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0a720644ed4dfd8b82b4fc0c507290, State: Initialized, Role: FOLLOWER
I20260925 10:30:53.666340 2368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260925 10:30:53.666630 2368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:53.666792 2368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:53.666985 2368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:53.669636 2368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260925 10:30:53.670202 2368 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0a720644ed4dfd8b82b4fc0c507290; no voters:
I20260925 10:30:53.670949 2368 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:53.671061 2378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:53.673978 2368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:53.673817 2378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [term 1 LEADER]: Becoming Leader. State: Replica: 6e0a720644ed4dfd8b82b4fc0c507290, State: Running, Role: LEADER
I20260925 10:30:53.674793 2378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } }
I20260925 10:30:53.678573 2379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } } }
I20260925 10:30:53.678685 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0a720644ed4dfd8b82b4fc0c507290. Latest consensus state: current_term: 1 leader_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0a720644ed4dfd8b82b4fc0c507290" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45665 } } }
I20260925 10:30:53.679031 2379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:53.679080 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:53.681943 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:53.664834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:34643 (local address 127.0.0.1:45665)
0925 10:30:53.665067 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:53.665074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:53.665513 (+ 439us) server_negotiation.cc:407] Connection header received
0925 10:30:53.665719 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:53.665724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:53.665815 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:53.665925 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:53.667525 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:53.668356 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:53.669653 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:53.670159 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:53.670299 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:53.673125 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:53.673131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:53.673135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:53.673172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:53.676854 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:53.677977 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:53.677983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:53.677991 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:53.678149 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:53.678853 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:53.678862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:53.678865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:53.679665 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:53.681008 (+ 1343us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:53.681188 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 10:30:53.681416 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:30:53.690847 2387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:53.696395 2387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:53.704850 2387 catalog_manager.cc:1347] Generated new cluster ID: bec1de6f05434425bfc2c1d38e7072a9
I20260925 10:30:53.704993 2387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:53.722997 2387 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:53.724304 2387 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:53.731922 2387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e0a720644ed4dfd8b82b4fc0c507290: Generated new TSK 0
I20260925 10:30:53.732956 2387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:53.747571 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2301
I20260925 10:30:53.748850 2363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:53.842839 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2301
W20260925 10:30:53.860512 2286 connection.cc:535] client connection to 127.0.0.1:45665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45665): Connection reset by peer (error 104)
2026-09-25T10:30:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2400](info): set up 2 sockets
Sep 25 10:30:53 dist-test-slave-f9rl krb5kdc[2400](info): commencing operation
krb5kdc: starting...
W20260925 10:30:55.003264 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:55 dist-test-slave-f9rl krb5kdc[2400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:55.272658 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:55.612805 2413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:55.613101 2413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:55.613273 2413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:55.620679 2413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:55.620798 2413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:55.620906 2413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:55.621016 2413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:55.621093 2413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:55.631114 2413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:55.632439 2413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:55.633831 2413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:55.645913 2418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:55.645897 2421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:55.646392 2419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:55.646771 2413 server_base.cc:1034] running on GCE node
I20260925 10:30:55.647594 2413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:55.649101 2413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:55.650328 2413 hybrid_clock.cc:648] HybridClock initialized: now 1790332255650284 us; error 40 us; skew 500 ppm
I20260925 10:30:55.650701 2413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:55 dist-test-slave-f9rl krb5kdc[2400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:55.661824 2413 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:55.663519 2413 webserver.cc:460] Webserver started at http://127.0.0.1:41583/ using document root <none> and password file <none>
I20260925 10:30:55.664276 2413 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:55.664425 2413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:55.664737 2413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:55.667913 2413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "eb61a86100c14d6298b77c43e749db13"
format_stamp: "Formatted at 2026-09-25 10:30:55 on dist-test-slave-f9rl"
server_key: "8df4d50a3f0c3987d795c250ea4bfd95"
server_key_iv: "5444c15e9e46b55cf203491ec78ee20f"
server_key_version: "encryptionkey@0"
I20260925 10:30:55.668737 2413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "eb61a86100c14d6298b77c43e749db13"
format_stamp: "Formatted at 2026-09-25 10:30:55 on dist-test-slave-f9rl"
server_key: "8df4d50a3f0c3987d795c250ea4bfd95"
server_key_iv: "5444c15e9e46b55cf203491ec78ee20f"
server_key_version: "encryptionkey@0"
I20260925 10:30:55.674055 2413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:30:55.677637 2428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:55.678994 2413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:30:55.679169 2413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "eb61a86100c14d6298b77c43e749db13"
format_stamp: "Formatted at 2026-09-25 10:30:55 on dist-test-slave-f9rl"
server_key: "8df4d50a3f0c3987d795c250ea4bfd95"
server_key_iv: "5444c15e9e46b55cf203491ec78ee20f"
server_key_version: "encryptionkey@0"
I20260925 10:30:55.679370 2413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:55.701251 2413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:55.705502 2413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:55.705794 2413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:55.724818 2413 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36745
I20260925 10:30:55.724841 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36745 every 8 connection(s)
I20260925 10:30:55.726353 2413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:55.729571 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2413
I20260925 10:30:55.729771 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:55.731003 2480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:55.734732 13281 external_mini_cluster.cc:1428] Setting key a7deff20152613adfdbfe87ac061d7bf
I20260925 10:30:55.743263 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13: Bootstrap starting.
I20260925 10:30:55.747051 2480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:55.748256 2480 log.cc:824] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:55.748440 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:55.736588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:44839 (local address 127.0.0.1:36745)
0925 10:30:55.737379 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:55.737391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:55.737477 (+ 86us) server_negotiation.cc:407] Connection header received
0925 10:30:55.739207 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:55.739231 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:55.739671 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:55.740017 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:55.740618 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:55.742031 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:55.743882 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:55.744582 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:55.744716 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:55.746113 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:55.746120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:55.746134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:55.746696 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:55.746767 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:55.746792 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:55.746913 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:55.747258 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 10:30:55.747428 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":357,"threads_started":1}
I20260925 10:30:55.751282 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13: No bootstrap required, opened a new log
I20260925 10:30:55.752310 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:55.759924 2480 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } }
I20260925 10:30:55.760316 2480 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:55.760427 2480 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb61a86100c14d6298b77c43e749db13, State: Initialized, Role: FOLLOWER
I20260925 10:30:55.761067 2480 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } }
I20260925 10:30:55.761329 2480 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:55.761502 2480 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:55.761693 2480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:55.763378 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:55.756678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:60907 (local address 127.0.0.1:36745)
0925 10:30:55.756856 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:55.756863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:55.757401 (+ 538us) server_negotiation.cc:407] Connection header received
0925 10:30:55.757592 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:55.757601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:55.757683 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:55.757792 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:55.758263 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:55.759394 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:55.760829 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:55.761321 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:55.761473 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:55.762624 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:55.762630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:55.762634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:55.762661 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:55.762712 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:55.762727 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:55.762864 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:55.763049 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:30:55.763109 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:30:55.764263 2480 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } }
I20260925 10:30:55.764844 2480 leader_election.cc:302] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb61a86100c14d6298b77c43e749db13; no voters:
I20260925 10:30:55.765494 2480 leader_election.cc:288] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:55.765614 2490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:55.766294 2490 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [term 1 LEADER]: Becoming Leader. State: Replica: eb61a86100c14d6298b77c43e749db13, State: Running, Role: LEADER
I20260925 10:30:55.766876 2490 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } }
I20260925 10:30:55.767627 2480 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:55.769764 2492 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb61a86100c14d6298b77c43e749db13. Latest consensus state: current_term: 1 leader_uuid: "eb61a86100c14d6298b77c43e749db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } } }
I20260925 10:30:55.770200 2492 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:55.774997 2491 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb61a86100c14d6298b77c43e749db13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb61a86100c14d6298b77c43e749db13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36745 } } }
I20260925 10:30:55.775429 2491 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:55.776280 2496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:55.781689 2496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:55.792090 2496 catalog_manager.cc:1347] Generated new cluster ID: f2910cb6bd7c4779a4c4a3862ee815e6
I20260925 10:30:55.792236 2496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:55.803161 2496 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:55.804630 2496 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:55.818589 2496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb61a86100c14d6298b77c43e749db13: Generated new TSK 0
I20260925 10:30:55.819865 2496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:55.829802 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2413
W20260925 10:30:55.830870 2475 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.116:44839
W20260925 10:30:55.831656 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2413: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:55.831809 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2413
W20260925 10:30:55.850121 2398 connection.cc:535] client connection to 127.0.0.1:36745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36745): Connection reset by peer (error 104)
2026-09-25T10:30:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:55 dist-test-slave-f9rl krb5kdc[2511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:55 dist-test-slave-f9rl krb5kdc[2511](info): set up 2 sockets
Sep 25 10:30:55 dist-test-slave-f9rl krb5kdc[2511](info): commencing operation
krb5kdc: starting...
W20260925 10:30:56.959374 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:57 dist-test-slave-f9rl krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:30:57.225101 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:57.560702 2524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:57.561062 2524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:57.561232 2524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:57.568718 2524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:57.568836 2524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:57.568939 2524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:57.569038 2524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:57.569113 2524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:57.579216 2524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:57.580540 2524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:57.581892 2524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:57.593827 2530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:57.593870 2529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:57.593895 2532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:57.594453 2524 server_base.cc:1034] running on GCE node
I20260925 10:30:57.595285 2524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:57.596791 2524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:57.598042 2524 hybrid_clock.cc:648] HybridClock initialized: now 1790332257597992 us; error 48 us; skew 500 ppm
I20260925 10:30:57.598402 2524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:57 dist-test-slave-f9rl krb5kdc[2511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:57.601590 2524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:30:57.603039 2524 webserver.cc:460] Webserver started at http://127.0.0.1:42407/ using document root <none> and password file <none>
I20260925 10:30:57.603763 2524 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:57.603907 2524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:57.604235 2524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:57.607357 2524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347"
format_stamp: "Formatted at 2026-09-25 10:30:57 on dist-test-slave-f9rl"
server_key: "c2a7277e2a1683e164ab3124e867cc74"
server_key_iv: "6d4a2769ec9b5e6668bd6fe0cee940c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:57.608116 2524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347"
format_stamp: "Formatted at 2026-09-25 10:30:57 on dist-test-slave-f9rl"
server_key: "c2a7277e2a1683e164ab3124e867cc74"
server_key_iv: "6d4a2769ec9b5e6668bd6fe0cee940c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:57.613185 2524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 10:30:57.616518 2539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:57.617910 2524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:30:57.618099 2524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347"
format_stamp: "Formatted at 2026-09-25 10:30:57 on dist-test-slave-f9rl"
server_key: "c2a7277e2a1683e164ab3124e867cc74"
server_key_iv: "6d4a2769ec9b5e6668bd6fe0cee940c2"
server_key_version: "encryptionkey@0"
I20260925 10:30:57.618305 2524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:57.623974 2524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:57.627988 2524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:57.628307 2524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:57.647158 2524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45011
I20260925 10:30:57.647203 2590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45011 every 8 connection(s)
I20260925 10:30:57.648582 2524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:57.650007 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2524
I20260925 10:30:57.650192 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:57.650731 13281 external_mini_cluster.cc:1428] Setting key e88d0d54003ca9cb4e811b0ec24de65e
I20260925 10:30:57.653157 2591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:57.658384 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:57.652502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:49107 (local address 127.0.0.1:45011)
0925 10:30:57.653350 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:57.653362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:57.653401 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:30:57.655034 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:57.655055 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:57.655409 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:57.655744 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:57.656791 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:57.656797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:57.656812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:57.656826 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:57.656901 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:57.656920 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:57.657085 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:57.657223 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:30:57.657422 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":361,"threads_started":1}
I20260925 10:30:57.661912 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:57.665913 2591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347: Bootstrap starting.
I20260925 10:30:57.669692 2591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:57.669956 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:57.667311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.116:34961 (local address 127.0.0.1:45011)
0925 10:30:57.667512 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:57.667518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:57.667954 (+ 436us) server_negotiation.cc:407] Connection header received
0925 10:30:57.668113 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:57.668122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:57.668182 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:57.668289 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:57.669175 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:57.669181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:57.669185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:30:57.669204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:57.669253 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:30:57.669272 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:57.669388 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:57.669513 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:30:57.669583 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 10:30:57.670888 2591 log.cc:824] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347: Log is configured to *not* fsync() on all Append() calls
I20260925 10:30:57.673686 2591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347: No bootstrap required, opened a new log
I20260925 10:30:57.680868 2591 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 10:30:57.681245 2591 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:57.681368 2591 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc7fda2e9f74b94b4e8b7e4f44ff347, State: Initialized, Role: FOLLOWER
I20260925 10:30:57.681954 2591 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 10:30:57.682189 2591 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:57.682348 2591 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:57.682520 2591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:57.684748 2591 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 10:30:57.685240 2591 leader_election.cc:302] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbc7fda2e9f74b94b4e8b7e4f44ff347; no voters:
I20260925 10:30:57.685850 2591 leader_election.cc:288] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:57.685982 2601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:57.686683 2601 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [term 1 LEADER]: Becoming Leader. State: Replica: bbc7fda2e9f74b94b4e8b7e4f44ff347, State: Running, Role: LEADER
I20260925 10:30:57.687258 2601 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 10:30:57.687978 2591 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:57.690428 2602 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } } }
I20260925 10:30:57.690950 2602 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:57.691162 2603 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbc7fda2e9f74b94b4e8b7e4f44ff347. Latest consensus state: current_term: 1 leader_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbc7fda2e9f74b94b4e8b7e4f44ff347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } } }
I20260925 10:30:57.691416 2603 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:57.694059 2610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:57.698948 2610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:57.708006 2610 catalog_manager.cc:1347] Generated new cluster ID: 6b1cd118ecc34074a5b1bfaa79582329
I20260925 10:30:57.708137 2610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:57.722522 2610 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:57.723153 2610 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:57.729856 2610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbc7fda2e9f74b94b4e8b7e4f44ff347: Generated new TSK 0
I20260925 10:30:57.730712 2610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:57.747047 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2524
W20260925 10:30:57.748756 2586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.116:49107
W20260925 10:30:57.749598 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2524: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:30:57.749835 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2524
2026-09-25T10:30:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:57 dist-test-slave-f9rl krb5kdc[2622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:57 dist-test-slave-f9rl krb5kdc[2622](info): set up 2 sockets
Sep 25 10:30:57 dist-test-slave-f9rl krb5kdc[2622](info): commencing operation
krb5kdc: starting...
W20260925 10:30:58.879185 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:30:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:30:59.151226 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40949
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:34381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:40949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:30:59.500523 2635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:30:59.500835 2635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:30:59.501008 2635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:30:59.508347 2635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:30:59.508467 2635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:30:59.508585 2635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:30:59.508675 2635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:30:59.508778 2635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:30:59.518977 2635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:34381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:40949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:40949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:30:59.520505 2635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:30:59.521936 2635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:30:59.535043 2640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:59.535244 2643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:30:59.535250 2641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:30:59.536451 2635 server_base.cc:1034] running on GCE node
I20260925 10:30:59.537493 2635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:30:59.539184 2635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:30:59.540447 2635 hybrid_clock.cc:648] HybridClock initialized: now 1790332259540404 us; error 52 us; skew 500 ppm
I20260925 10:30:59.540791 2635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332259, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:30:59.543972 2635 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:30:59.545426 2635 webserver.cc:460] Webserver started at http://127.12.248.126:46279/ using document root <none> and password file <none>
I20260925 10:30:59.546151 2635 fs_manager.cc:359] Metadata directory not provided
I20260925 10:30:59.546254 2635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:30:59.546526 2635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:30:59.549758 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "135056cc474f4e729036460df62277e5"
format_stamp: "Formatted at 2026-09-25 10:30:59 on dist-test-slave-f9rl"
server_key: "ba6d2a8d531c390e090fd37f08af2120"
server_key_iv: "7b4106d84c491a3ffbba5a8b48f06a38"
server_key_version: "encryptionkey@0"
I20260925 10:30:59.550474 2635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "135056cc474f4e729036460df62277e5"
format_stamp: "Formatted at 2026-09-25 10:30:59 on dist-test-slave-f9rl"
server_key: "ba6d2a8d531c390e090fd37f08af2120"
server_key_iv: "7b4106d84c491a3ffbba5a8b48f06a38"
server_key_version: "encryptionkey@0"
I20260925 10:30:59.555837 2635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:30:59.560058 2650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:30:59.561897 2635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:30:59.562068 2635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "135056cc474f4e729036460df62277e5"
format_stamp: "Formatted at 2026-09-25 10:30:59 on dist-test-slave-f9rl"
server_key: "ba6d2a8d531c390e090fd37f08af2120"
server_key_iv: "7b4106d84c491a3ffbba5a8b48f06a38"
server_key_version: "encryptionkey@0"
I20260925 10:30:59.562264 2635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:30:59.576825 2635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:30:59.580912 2635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:30:59.581156 2635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:30:59.600219 2635 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:40949
I20260925 10:30:59.600221 2701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:40949 every 8 connection(s)
I20260925 10:30:59.601969 2635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:30:59.605902 2702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:30:59.609090 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2635
I20260925 10:30:59.609293 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:30:59.609930 13281 external_mini_cluster.cc:1428] Setting key 904700a7793613242325f95522850b0a
I20260925 10:30:59.619760 2702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5: Bootstrap starting.
I20260925 10:30:59.623520 2702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5: Neither blocks nor log segments found. Creating new log.
I20260925 10:30:59.624565 2702 log.cc:824] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:30:59.627595 2702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5: No bootstrap required, opened a new log
I20260925 10:30:59.634925 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:59.612214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57077 (local address 127.12.248.126:40949)
0925 10:30:59.613189 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:59.613206 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:59.613644 (+ 438us) server_negotiation.cc:407] Connection header received
0925 10:30:59.615522 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:59.615554 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:59.616115 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:59.616572 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:59.618156 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:59.620098 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:59.621404 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:59.622465 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:59.622645 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:59.627177 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:59.627186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:59.627200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:59.627243 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:59.630556 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:59.631396 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:59.631402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:59.631408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:59.631580 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:59.632066 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:59.632073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:59.632075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:59.632617 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:59.632994 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:59.633531 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 10:30:59.633881 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":442,"threads_started":1}
I20260925 10:30:59.635673 2702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } }
I20260925 10:30:59.636034 2702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:30:59.636134 2702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135056cc474f4e729036460df62277e5, State: Initialized, Role: FOLLOWER
I20260925 10:30:59.636726 2702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } }
I20260925 10:30:59.636988 2702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:30:59.637142 2702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:30:59.637334 2702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:30:59.639140 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:30:59.640002 2702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } }
I20260925 10:30:59.640482 2702 leader_election.cc:302] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135056cc474f4e729036460df62277e5; no voters:
I20260925 10:30:59.641085 2702 leader_election.cc:288] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:30:59.641281 2707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:30:59.643067 2702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:30:59.653201 2707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [term 1 LEADER]: Becoming Leader. State: Replica: 135056cc474f4e729036460df62277e5, State: Running, Role: LEADER
I20260925 10:30:59.654033 2707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } }
I20260925 10:30:59.657954 2708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135056cc474f4e729036460df62277e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } } }
I20260925 10:30:59.658376 2708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:59.661546 2710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135056cc474f4e729036460df62277e5. Latest consensus state: current_term: 1 leader_uuid: "135056cc474f4e729036460df62277e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135056cc474f4e729036460df62277e5" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 40949 } } }
I20260925 10:30:59.662017 2710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:30:59.663236 2727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:30:59.669242 2727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:30:59.672911 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:30:59.655124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50043 (local address 127.12.248.126:40949)
0925 10:30:59.655318 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:30:59.655324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:30:59.656003 (+ 679us) server_negotiation.cc:407] Connection header received
0925 10:30:59.656228 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:30:59.656232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:30:59.656305 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:30:59.656438 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:30:59.658689 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:59.660273 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:59.662858 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:30:59.663636 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:30:59.663764 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:30:59.666648 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:30:59.666656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:30:59.666660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:30:59.666700 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:30:59.669889 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:59.670699 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:59.670703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:59.670706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:59.670832 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:30:59.671324 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:30:59.671328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:30:59.671330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:30:59.671640 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:30:59.671928 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:30:59.672329 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 10:30:59.672514 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:30:59.678953 2727 catalog_manager.cc:1347] Generated new cluster ID: 3019507be78d43078e5eea7324d210a4
I20260925 10:30:59.679080 2727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:30:59.706233 2727 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:30:59.708241 2727 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:30:59.720558 2727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135056cc474f4e729036460df62277e5: Generated new TSK 0
I20260925 10:30:59.721675 2727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:30:59.730129 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2635
I20260925 10:30:59.731281 2697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:30:59.825486 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2635
W20260925 10:30:59.843003 2620 connection.cc:535] client connection to 127.12.248.126:40949 recv error: Network error: failed to read from TLS socket (remote: 127.12.248.126:40949): Connection reset by peer (error 104)
2026-09-25T10:30:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2734](info): set up 2 sockets
Sep 25 10:30:59 dist-test-slave-f9rl krb5kdc[2734](info): commencing operation
krb5kdc: starting...
W20260925 10:31:00.943418 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:01 dist-test-slave-f9rl krb5kdc[2734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:01.217758 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37947
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:01.557559 2747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:01.557914 2747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:01.558084 2747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:01.565603 2747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:01.565775 2747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:01.565887 2747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:01.565997 2747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:01.566074 2747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:01.576356 2747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:01.577827 2747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:01.579182 2747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:01.591653 2752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:01.591588 2753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:01.591676 2755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:01.592911 2747 server_base.cc:1034] running on GCE node
I20260925 10:31:01.593948 2747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:01.595521 2747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:01.596808 2747 hybrid_clock.cc:648] HybridClock initialized: now 1790332261596754 us; error 46 us; skew 500 ppm
I20260925 10:31:01.597282 2747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:01 dist-test-slave-f9rl krb5kdc[2734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332261, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:01.606742 2747 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:01.608284 2747 webserver.cc:460] Webserver started at http://127.12.248.126:40837/ using document root <none> and password file <none>
I20260925 10:31:01.608982 2747 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:01.609084 2747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:01.609355 2747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:01.612479 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "99c3c5176d784bb38d52cdb230c06c16"
format_stamp: "Formatted at 2026-09-25 10:31:01 on dist-test-slave-f9rl"
server_key: "ae1aae1e27de9f6dc235d6add554b051"
server_key_iv: "94b91bf5fa80d88abf92ddb97c289e37"
server_key_version: "encryptionkey@0"
I20260925 10:31:01.613294 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "99c3c5176d784bb38d52cdb230c06c16"
format_stamp: "Formatted at 2026-09-25 10:31:01 on dist-test-slave-f9rl"
server_key: "ae1aae1e27de9f6dc235d6add554b051"
server_key_iv: "94b91bf5fa80d88abf92ddb97c289e37"
server_key_version: "encryptionkey@0"
I20260925 10:31:01.618788 2747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:31:01.622535 2762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:01.623999 2747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:31:01.624151 2747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "99c3c5176d784bb38d52cdb230c06c16"
format_stamp: "Formatted at 2026-09-25 10:31:01 on dist-test-slave-f9rl"
server_key: "ae1aae1e27de9f6dc235d6add554b051"
server_key_iv: "94b91bf5fa80d88abf92ddb97c289e37"
server_key_version: "encryptionkey@0"
I20260925 10:31:01.624325 2747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:01.638411 2747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:01.643303 2747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:01.643589 2747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:01.663758 2747 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37947
I20260925 10:31:01.663894 2814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37947 every 8 connection(s)
I20260925 10:31:01.665215 2747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:01.669296 2815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:01.674281 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2747
I20260925 10:31:01.674477 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:01.675063 13281 external_mini_cluster.cc:1428] Setting key 843084340df4b547e81ffc87ff7e9a7b
I20260925 10:31:01.680989 2815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16: Bootstrap starting.
I20260925 10:31:01.684247 2815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:01.685384 2815 log.cc:824] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:01.689052 2815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16: No bootstrap required, opened a new log
I20260925 10:31:01.689074 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:01.676920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36919 (local address 127.12.248.126:37947)
0925 10:31:01.677826 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:01.677838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:01.677886 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:31:01.679684 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:01.679709 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:01.680122 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:01.680522 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:01.681117 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:01.682518 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:01.683859 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:01.684616 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:01.684836 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:01.686244 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:01.686252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:01.686267 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:01.686950 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:01.687045 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:01.687084 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:01.687255 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:01.687530 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:31:01.687726 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":438,"threads_started":1}
I20260925 10:31:01.693186 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:01.697921 2815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } }
I20260925 10:31:01.698494 2815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:01.698625 2815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c3c5176d784bb38d52cdb230c06c16, State: Initialized, Role: FOLLOWER
I20260925 10:31:01.699381 2815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } }
I20260925 10:31:01.699718 2815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:01.699877 2815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:01.700084 2815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:01.703305 2815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } }
I20260925 10:31:01.703977 2815 leader_election.cc:302] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99c3c5176d784bb38d52cdb230c06c16; no voters:
I20260925 10:31:01.705170 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:01.697694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54897 (local address 127.12.248.126:37947)
0925 10:31:01.697926 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:01.697935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:01.698445 (+ 510us) server_negotiation.cc:407] Connection header received
0925 10:31:01.698660 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:01.698666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:01.698754 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:01.698878 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:01.699512 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:01.700537 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:01.702438 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:01.702884 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:01.702976 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:01.704279 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:01.704283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:01.704286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:01.704305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:01.704352 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:01.704367 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:01.704476 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:01.704761 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:31:01.704803 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:31:01.705298 2815 leader_election.cc:288] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:01.705406 2825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:01.706099 2825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [term 1 LEADER]: Becoming Leader. State: Replica: 99c3c5176d784bb38d52cdb230c06c16, State: Running, Role: LEADER
I20260925 10:31:01.706943 2825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } }
I20260925 10:31:01.708719 2815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:01.710090 2827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c3c5176d784bb38d52cdb230c06c16. Latest consensus state: current_term: 1 leader_uuid: "99c3c5176d784bb38d52cdb230c06c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } } }
I20260925 10:31:01.710496 2827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:01.711181 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99c3c5176d784bb38d52cdb230c06c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c3c5176d784bb38d52cdb230c06c16" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37947 } } }
I20260925 10:31:01.711455 2830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:01.711534 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:01.724776 2830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:01.734395 2830 catalog_manager.cc:1347] Generated new cluster ID: acedb2250fbd476f85535bd3fb2c8263
I20260925 10:31:01.734540 2830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:01.754805 2830 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:01.756183 2830 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:01.768368 2830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c3c5176d784bb38d52cdb230c06c16: Generated new TSK 0
I20260925 10:31:01.769253 2830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:01.777531 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2747
W20260925 10:31:01.778875 2810 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36919
W20260925 10:31:01.779615 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2747: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:31:01.779767 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2747
W20260925 10:31:01.796953 2732 connection.cc:535] client connection to 127.12.248.126:37947 recv error: Network error: failed to read from TLS socket (remote: 127.12.248.126:37947): Connection reset by peer (error 104)
2026-09-25T10:31:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:01 dist-test-slave-f9rl krb5kdc[2846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:01 dist-test-slave-f9rl krb5kdc[2846](info): set up 2 sockets
Sep 25 10:31:01 dist-test-slave-f9rl krb5kdc[2846](info): commencing operation
krb5kdc: starting...
W20260925 10:31:02.895431 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:03 dist-test-slave-f9rl krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:03.169766 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37713
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:37713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:03.506991 2859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:03.507292 2859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:03.507475 2859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:03.514936 2859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:03.515080 2859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:03.515200 2859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:03.515321 2859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:03.515384 2859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:03.525296 2859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:37713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:37713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:03.526715 2859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:03.528060 2859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:03.539881 2864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:03.539937 2867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:03.539912 2865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:03.541095 2859 server_base.cc:1034] running on GCE node
I20260925 10:31:03.541971 2859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:03.543380 2859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:03.544620 2859 hybrid_clock.cc:648] HybridClock initialized: now 1790332263544565 us; error 51 us; skew 500 ppm
I20260925 10:31:03.544994 2859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:03 dist-test-slave-f9rl krb5kdc[2846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332263, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:03.548347 2859 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:03.550086 2859 webserver.cc:460] Webserver started at http://127.12.248.126:40451/ using document root <none> and password file <none>
I20260925 10:31:03.550740 2859 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:03.550840 2859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:03.551090 2859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:03.554088 2859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "4d3e736d47954709b1092fa48364eabd"
format_stamp: "Formatted at 2026-09-25 10:31:03 on dist-test-slave-f9rl"
server_key: "5f514e1e593264c8655366c94a298fee"
server_key_iv: "0c7ab78fd75b66f16a787917aadfdad2"
server_key_version: "encryptionkey@0"
I20260925 10:31:03.554811 2859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "4d3e736d47954709b1092fa48364eabd"
format_stamp: "Formatted at 2026-09-25 10:31:03 on dist-test-slave-f9rl"
server_key: "5f514e1e593264c8655366c94a298fee"
server_key_iv: "0c7ab78fd75b66f16a787917aadfdad2"
server_key_version: "encryptionkey@0"
I20260925 10:31:03.560110 2859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:31:03.563589 2874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:03.565116 2859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:31:03.565285 2859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "4d3e736d47954709b1092fa48364eabd"
format_stamp: "Formatted at 2026-09-25 10:31:03 on dist-test-slave-f9rl"
server_key: "5f514e1e593264c8655366c94a298fee"
server_key_iv: "0c7ab78fd75b66f16a787917aadfdad2"
server_key_version: "encryptionkey@0"
I20260925 10:31:03.565578 2859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:03.571446 2859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:03.575886 2859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:03.576241 2859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:03.595430 2859 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:37713
I20260925 10:31:03.595454 2925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:37713 every 8 connection(s)
I20260925 10:31:03.596865 2859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:03.600620 2926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:03.605309 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2859
I20260925 10:31:03.605540 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:03.606104 13281 external_mini_cluster.cc:1428] Setting key 757b643473184ee24f794ce36003a5c4
I20260925 10:31:03.612499 2926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd: Bootstrap starting.
I20260925 10:31:03.614452 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:03.607979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54953 (local address 127.12.248.126:37713)
0925 10:31:03.608807 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:03.608821 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:03.608865 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:31:03.610909 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:03.610933 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:03.611273 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:03.611586 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:03.612747 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:03.612753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:03.612766 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:03.612782 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:03.612850 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:03.612870 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:03.613062 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:03.613179 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:31:03.613405 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":382,"threads_started":1}
I20260925 10:31:03.615919 2926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:03.617097 2926 log.cc:824] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:03.618340 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:03.620077 2926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd: No bootstrap required, opened a new log
I20260925 10:31:03.625366 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:03.622917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45763 (local address 127.12.248.126:37713)
0925 10:31:03.623104 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:03.623110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:03.623453 (+ 343us) server_negotiation.cc:407] Connection header received
0925 10:31:03.623574 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:03.623578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:03.623642 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:03.623769 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:03.624674 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:03.624680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:03.624685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:03.624703 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:03.624755 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:03.624776 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:03.624911 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:03.624982 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:31:03.625045 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:31:03.629297 2926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } }
I20260925 10:31:03.629726 2926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:03.629807 2926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3e736d47954709b1092fa48364eabd, State: Initialized, Role: FOLLOWER
I20260925 10:31:03.630326 2926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } }
I20260925 10:31:03.630555 2926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:03.630661 2926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:03.630791 2926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:03.632958 2926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } }
I20260925 10:31:03.633477 2926 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3e736d47954709b1092fa48364eabd; no voters:
I20260925 10:31:03.634217 2926 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:03.634352 2936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:03.635031 2936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [term 1 LEADER]: Becoming Leader. State: Replica: 4d3e736d47954709b1092fa48364eabd, State: Running, Role: LEADER
I20260925 10:31:03.635582 2936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } }
I20260925 10:31:03.636361 2926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:03.638640 2937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d3e736d47954709b1092fa48364eabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } } }
I20260925 10:31:03.639178 2937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:03.639979 2938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3e736d47954709b1092fa48364eabd. Latest consensus state: current_term: 1 leader_uuid: "4d3e736d47954709b1092fa48364eabd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3e736d47954709b1092fa48364eabd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 37713 } } }
I20260925 10:31:03.640427 2938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:03.640789 2943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:03.646649 2943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:03.656478 2943 catalog_manager.cc:1347] Generated new cluster ID: 329133cce9cc4cf68822d408d46a8c67
I20260925 10:31:03.656603 2943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:03.689637 2943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:03.690145 2943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:03.702873 2943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d3e736d47954709b1092fa48364eabd: Generated new TSK 0
I20260925 10:31:03.703926 2943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:03.710536 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2859
W20260925 10:31:03.711944 2921 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54953
W20260925 10:31:03.712631 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2859: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:31:03.712795 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2859
2026-09-25T10:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:03 dist-test-slave-f9rl krb5kdc[2957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:03 dist-test-slave-f9rl krb5kdc[2957](info): set up 2 sockets
Sep 25 10:31:03 dist-test-slave-f9rl krb5kdc[2957](info): commencing operation
krb5kdc: starting...
W20260925 10:31:04.839474 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[2957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:05.109830 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:45763
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:05.449604 2970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:05.449923 2970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:05.450086 2970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:05.457418 2970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:05.457633 2970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:05.457733 2970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:05.457854 2970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:05.457933 2970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:05.468048 2970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:45763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:05.469475 2970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:05.470870 2970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:05.482883 2976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:05.482810 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:05.482779 2975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:05.484098 2970 server_base.cc:1034] running on GCE node
I20260925 10:31:05.484862 2970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:05.486402 2970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:05.487620 2970 hybrid_clock.cc:648] HybridClock initialized: now 1790332265487582 us; error 33 us; skew 500 ppm
I20260925 10:31:05.488000 2970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[2957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332265, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:05.491320 2970 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:05.492871 2970 webserver.cc:460] Webserver started at http://127.12.248.126:39635/ using document root <none> and password file <none>
I20260925 10:31:05.493613 2970 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:05.493723 2970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:05.493990 2970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:05.496973 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "9830956fa1db464fbf19ad3ab22ce71c"
format_stamp: "Formatted at 2026-09-25 10:31:05 on dist-test-slave-f9rl"
server_key: "ba5c05124a9594b4e5bbdc877623f3dc"
server_key_iv: "e30941261c4012023ba4827cf2cea705"
server_key_version: "encryptionkey@0"
I20260925 10:31:05.497776 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "9830956fa1db464fbf19ad3ab22ce71c"
format_stamp: "Formatted at 2026-09-25 10:31:05 on dist-test-slave-f9rl"
server_key: "ba5c05124a9594b4e5bbdc877623f3dc"
server_key_iv: "e30941261c4012023ba4827cf2cea705"
server_key_version: "encryptionkey@0"
I20260925 10:31:05.502993 2970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 10:31:05.506829 2985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:05.508343 2970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 10:31:05.508554 2970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "9830956fa1db464fbf19ad3ab22ce71c"
format_stamp: "Formatted at 2026-09-25 10:31:05 on dist-test-slave-f9rl"
server_key: "ba5c05124a9594b4e5bbdc877623f3dc"
server_key_iv: "e30941261c4012023ba4827cf2cea705"
server_key_version: "encryptionkey@0"
I20260925 10:31:05.508777 2970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:05.530521 2970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:05.535171 2970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:05.535473 2970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:05.554438 2970 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:45763
I20260925 10:31:05.554469 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:45763 every 8 connection(s)
I20260925 10:31:05.555925 2970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:05.559824 3037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:05.565927 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 2970
I20260925 10:31:05.566129 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:05.566591 13281 external_mini_cluster.cc:1428] Setting key 90762f3860bfbe9ecf91f6ad5c09d9f6
I20260925 10:31:05.571369 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c: Bootstrap starting.
I20260925 10:31:05.575310 3037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:05.576602 3037 log.cc:824] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:05.580417 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c: No bootstrap required, opened a new log
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[2957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:31:05.588606 3037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } }
I20260925 10:31:05.589013 3037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:05.589138 3037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9830956fa1db464fbf19ad3ab22ce71c, State: Initialized, Role: FOLLOWER
I20260925 10:31:05.589809 3037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } }
I20260925 10:31:05.590118 3037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:05.590291 3037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:05.590479 3037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:05.590996 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:05.568452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42147 (local address 127.12.248.126:45763)
0925 10:31:05.569332 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:05.569344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:05.569392 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:31:05.571108 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:05.571131 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:05.571578 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:05.571925 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:05.573813 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:05.575158 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:05.576951 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:05.578084 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:05.578215 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:05.583196 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:05.583207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:05.583223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:05.583271 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:05.586524 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:05.587215 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:05.587222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:05.587228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:05.587393 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:05.587799 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:05.587813 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:05.587815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:05.588330 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:05.588625 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:05.589077 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 10:31:05.589743 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":424,"threads_started":1}
I20260925 10:31:05.592957 3037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } }
I20260925 10:31:05.593566 3037 leader_election.cc:302] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9830956fa1db464fbf19ad3ab22ce71c; no voters:
I20260925 10:31:05.594287 3037 leader_election.cc:288] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:05.594419 3042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:05.594887 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:05.595178 3042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [term 1 LEADER]: Becoming Leader. State: Replica: 9830956fa1db464fbf19ad3ab22ce71c, State: Running, Role: LEADER
I20260925 10:31:05.595700 3042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } }
I20260925 10:31:05.598162 3044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9830956fa1db464fbf19ad3ab22ce71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } } }
I20260925 10:31:05.598533 3044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:05.601224 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9830956fa1db464fbf19ad3ab22ce71c. Latest consensus state: current_term: 1 leader_uuid: "9830956fa1db464fbf19ad3ab22ce71c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9830956fa1db464fbf19ad3ab22ce71c" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 45763 } } }
I20260925 10:31:05.604004 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:05.603858 3037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:05.609308 3046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:05.616778 3046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:05.630654 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:05.611552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37673 (local address 127.12.248.126:45763)
0925 10:31:05.611713 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:05.611721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:05.612263 (+ 542us) server_negotiation.cc:407] Connection header received
0925 10:31:05.612562 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:05.612569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:05.612671 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:05.612804 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:05.614919 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:05.616051 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:05.617968 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:05.618645 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:05.618745 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:05.622361 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:05.622367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:05.622372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:05.622411 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:05.626294 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:05.627245 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:05.627253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:05.627276 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:05.627462 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:05.628021 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:05.628029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:05.628034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:05.628643 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:05.628957 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:05.629773 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 10:31:05.630101 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:31:05.632840 3065 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:31:05.635620 3046 catalog_manager.cc:1347] Generated new cluster ID: 78717da0e475466d9cb991ad85cff7c7
I20260925 10:31:05.636633 3046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:05.656424 3046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:05.657869 3046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:05.665895 3046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9830956fa1db464fbf19ad3ab22ce71c: Generated new TSK 0
I20260925 10:31:05.666787 3046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:05.676406 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 2970
I20260925 10:31:05.677604 3032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:05.771026 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 2970
2026-09-25T10:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[3070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[3070](info): set up 2 sockets
Sep 25 10:31:05 dist-test-slave-f9rl krb5kdc[3070](info): commencing operation
krb5kdc: starting...
W20260925 10:31:06.924103 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:07 dist-test-slave-f9rl krb5kdc[3070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:07.204288 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34553
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:07.538888 3083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:07.539188 3083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:07.539280 3083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:07.546551 3083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:07.546670 3083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:07.546716 3083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:07.546779 3083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:07.546829 3083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:07.556797 3083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:07.558104 3083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:07.559368 3083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:07.571468 3088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:07.571552 3089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:07.571506 3091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:07.572538 3083 server_base.cc:1034] running on GCE node
I20260925 10:31:07.573315 3083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:07.574796 3083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:07.576046 3083 hybrid_clock.cc:648] HybridClock initialized: now 1790332267576058 us; error 101 us; skew 500 ppm
I20260925 10:31:07.576454 3083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:07 dist-test-slave-f9rl krb5kdc[3070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332267, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:07.584389 3083 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:07.586102 3083 webserver.cc:460] Webserver started at http://127.12.248.126:45915/ using document root <none> and password file <none>
I20260925 10:31:07.586844 3083 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:07.586997 3083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:07.587347 3083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:07.590476 3083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "477a1769c5004f2f9ebd55b4403ecd54"
format_stamp: "Formatted at 2026-09-25 10:31:07 on dist-test-slave-f9rl"
server_key: "4ed79a8b93ef5a071594bb91b5e588ef"
server_key_iv: "18ca4acb65ddeb1dbb2566c4e283c746"
server_key_version: "encryptionkey@0"
I20260925 10:31:07.591272 3083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "477a1769c5004f2f9ebd55b4403ecd54"
format_stamp: "Formatted at 2026-09-25 10:31:07 on dist-test-slave-f9rl"
server_key: "4ed79a8b93ef5a071594bb91b5e588ef"
server_key_iv: "18ca4acb65ddeb1dbb2566c4e283c746"
server_key_version: "encryptionkey@0"
I20260925 10:31:07.596494 3083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:31:07.600237 3098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:07.601920 3083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:31:07.602113 3083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "477a1769c5004f2f9ebd55b4403ecd54"
format_stamp: "Formatted at 2026-09-25 10:31:07 on dist-test-slave-f9rl"
server_key: "4ed79a8b93ef5a071594bb91b5e588ef"
server_key_iv: "18ca4acb65ddeb1dbb2566c4e283c746"
server_key_version: "encryptionkey@0"
I20260925 10:31:07.602324 3083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:07.627784 3083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:07.632347 3083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:07.632673 3083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:07.651916 3083 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:34553
I20260925 10:31:07.651924 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:34553 every 8 connection(s)
I20260925 10:31:07.653350 3083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:07.657193 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:07.662571 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3083
I20260925 10:31:07.662768 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:07.663214 13281 external_mini_cluster.cc:1428] Setting key 64fdb0a1b9c5702d3fbe91bb9fcfa2c5
I20260925 10:31:07.669183 3150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54: Bootstrap starting.
I20260925 10:31:07.673291 3150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:07.674860 3150 log.cc:824] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:07.677738 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:07.665006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45375 (local address 127.12.248.126:34553)
0925 10:31:07.665843 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:07.665855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:07.665899 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:31:07.667675 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:07.667699 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:07.668124 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:07.668455 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:07.669140 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:07.670548 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:07.672696 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:07.673380 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:07.673656 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:07.675097 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:07.675105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:07.675119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:07.675699 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:07.675778 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:07.675804 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:07.675966 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:07.676227 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 10:31:07.676628 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":387,"threads_started":1}
I20260925 10:31:07.678247 3150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54: No bootstrap required, opened a new log
I20260925 10:31:07.682055 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:07.688529 3150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } }
I20260925 10:31:07.688982 3150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:07.689107 3150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477a1769c5004f2f9ebd55b4403ecd54, State: Initialized, Role: FOLLOWER
I20260925 10:31:07.689847 3150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } }
I20260925 10:31:07.690172 3150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:07.690315 3150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:07.690495 3150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:07.693236 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:07.686447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54245 (local address 127.12.248.126:34553)
0925 10:31:07.686617 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:07.686621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:07.686992 (+ 371us) server_negotiation.cc:407] Connection header received
0925 10:31:07.687181 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:07.687185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:07.687244 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:07.687322 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:07.688014 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:07.688743 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:07.690828 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:07.691266 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:07.691370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:07.692538 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:07.692542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:07.692545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:07.692565 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:07.692601 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:07.692615 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:07.692680 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:07.692856 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:31:07.692928 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:31:07.694011 3150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } }
I20260925 10:31:07.694571 3150 leader_election.cc:302] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 477a1769c5004f2f9ebd55b4403ecd54; no voters:
I20260925 10:31:07.695176 3150 leader_election.cc:288] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:07.695264 3160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:07.696077 3160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [term 1 LEADER]: Becoming Leader. State: Replica: 477a1769c5004f2f9ebd55b4403ecd54, State: Running, Role: LEADER
I20260925 10:31:07.696645 3160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } }
I20260925 10:31:07.697587 3150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:07.699870 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 477a1769c5004f2f9ebd55b4403ecd54. Latest consensus state: current_term: 1 leader_uuid: "477a1769c5004f2f9ebd55b4403ecd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } } }
I20260925 10:31:07.700847 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:07.699924 3161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "477a1769c5004f2f9ebd55b4403ecd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477a1769c5004f2f9ebd55b4403ecd54" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 34553 } } }
I20260925 10:31:07.702070 3161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:07.703394 3167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:07.709334 3167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:07.718806 3167 catalog_manager.cc:1347] Generated new cluster ID: b9f49a49ac614020af27b38ce49b098c
I20260925 10:31:07.718976 3167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:07.743930 3167 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:07.745481 3167 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:07.762727 3167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 477a1769c5004f2f9ebd55b4403ecd54: Generated new TSK 0
I20260925 10:31:07.763628 3167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:07.777897 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3083
W20260925 10:31:07.779021 3145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45375
W20260925 10:31:07.779701 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3083: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:31:07.779842 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3083
2026-09-25T10:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:07 dist-test-slave-f9rl krb5kdc[3181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:07 dist-test-slave-f9rl krb5kdc[3181](info): set up 2 sockets
Sep 25 10:31:07 dist-test-slave-f9rl krb5kdc[3181](info): commencing operation
krb5kdc: starting...
W20260925 10:31:08.911813 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:09 dist-test-slave-f9rl krb5kdc[3181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:09.184765 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36043
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:36263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:36043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:09.525147 3194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:09.525496 3194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:09.525676 3194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:09.533166 3194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:09.533293 3194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:09.533406 3194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:09.533511 3194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:09.533581 3194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:09.543756 3194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:36263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:36043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:36043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:09.545151 3194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:09.546595 3194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:09.558928 3202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:09.558965 3199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:09.559006 3200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:09.559777 3194 server_base.cc:1034] running on GCE node
I20260925 10:31:09.560658 3194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:09.562220 3194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:09.563457 3194 hybrid_clock.cc:648] HybridClock initialized: now 1790332269563403 us; error 46 us; skew 500 ppm
I20260925 10:31:09.563815 3194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:09 dist-test-slave-f9rl krb5kdc[3181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332269, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:09.567126 3194 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:09.568660 3194 webserver.cc:460] Webserver started at http://127.12.248.126:44969/ using document root <none> and password file <none>
I20260925 10:31:09.569371 3194 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:09.569545 3194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:09.569862 3194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:09.572964 3194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "8481b1eee6884467ac79444bf9362bcd"
format_stamp: "Formatted at 2026-09-25 10:31:09 on dist-test-slave-f9rl"
server_key: "4cdcdc3566f219cd71804502e3dcc491"
server_key_iv: "5e809f2d95f25570f00923e139c71116"
server_key_version: "encryptionkey@0"
I20260925 10:31:09.573812 3194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "8481b1eee6884467ac79444bf9362bcd"
format_stamp: "Formatted at 2026-09-25 10:31:09 on dist-test-slave-f9rl"
server_key: "4cdcdc3566f219cd71804502e3dcc491"
server_key_iv: "5e809f2d95f25570f00923e139c71116"
server_key_version: "encryptionkey@0"
I20260925 10:31:09.579080 3194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 10:31:09.582631 3209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:09.583964 3194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 10:31:09.584159 3194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "8481b1eee6884467ac79444bf9362bcd"
format_stamp: "Formatted at 2026-09-25 10:31:09 on dist-test-slave-f9rl"
server_key: "4cdcdc3566f219cd71804502e3dcc491"
server_key_iv: "5e809f2d95f25570f00923e139c71116"
server_key_version: "encryptionkey@0"
I20260925 10:31:09.584383 3194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:09.590111 3194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:09.594579 3194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:09.594913 3194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:09.613858 3194 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:36043
I20260925 10:31:09.613889 3260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:36043 every 8 connection(s)
I20260925 10:31:09.615314 3194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:09.619228 3261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:09.621701 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3194
I20260925 10:31:09.621881 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:09.622303 13281 external_mini_cluster.cc:1428] Setting key 66f6f61f4cd833e75baa6f28c9f6eebb
I20260925 10:31:09.630667 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:09.624256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40931 (local address 127.12.248.126:36043)
0925 10:31:09.625196 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:09.625207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:09.625250 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:31:09.626996 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:09.627020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:09.627389 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:09.627785 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:09.628959 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:09.628965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:09.628982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:09.628996 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:09.629060 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:09.629085 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:09.629262 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:09.629419 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 10:31:09.629679 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":408,"threads_started":1}
I20260925 10:31:09.631456 3261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: Bootstrap starting.
I20260925 10:31:09.634387 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:09.634775 3261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:09.635773 3261 log.cc:824] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:09.638877 3261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: No bootstrap required, opened a new log
I20260925 10:31:09.641607 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:09.639032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46811 (local address 127.12.248.126:36043)
0925 10:31:09.639245 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:09.639251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:09.639618 (+ 367us) server_negotiation.cc:407] Connection header received
0925 10:31:09.639792 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:09.639796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:09.639868 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:09.639949 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:09.640921 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:09.640925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:09.640928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:31:09.640942 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:09.640982 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:31:09.640996 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:09.641094 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:09.641217 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:31:09.641264 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:31:09.647152 3261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } }
I20260925 10:31:09.647547 3261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:31:09.647670 3261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8481b1eee6884467ac79444bf9362bcd, State: Initialized, Role: FOLLOWER
I20260925 10:31:09.648290 3261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } }
I20260925 10:31:09.648499 3261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:31:09.648625 3261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:31:09.648808 3261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:09.651156 3261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } }
I20260925 10:31:09.651665 3261 leader_election.cc:302] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8481b1eee6884467ac79444bf9362bcd; no voters:
I20260925 10:31:09.652275 3261 leader_election.cc:288] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:31:09.652468 3271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:09.653757 3271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [term 1 LEADER]: Becoming Leader. State: Replica: 8481b1eee6884467ac79444bf9362bcd, State: Running, Role: LEADER
I20260925 10:31:09.654321 3271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } }
I20260925 10:31:09.654584 3261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:09.657574 3272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8481b1eee6884467ac79444bf9362bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } } }
I20260925 10:31:09.657949 3272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:09.657256 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8481b1eee6884467ac79444bf9362bcd. Latest consensus state: current_term: 1 leader_uuid: "8481b1eee6884467ac79444bf9362bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481b1eee6884467ac79444bf9362bcd" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 36043 } } }
I20260925 10:31:09.658465 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd [sys.catalog]: This master's current role is: LEADER
W20260925 10:31:09.666565 3287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:31:09.666715 3287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:31:09.666893 3281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:09.669098 3281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:09.678172 3281 catalog_manager.cc:1347] Generated new cluster ID: e3653eabc1a0474aaf5ffdf676e0e783
I20260925 10:31:09.678344 3281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:09.709864 3281 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:09.710307 3281 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:09.718050 3281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8481b1eee6884467ac79444bf9362bcd: Generated new TSK 0
I20260925 10:31:09.718930 3281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:09.735268 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3194
W20260925 10:31:09.736634 3256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40931
W20260925 10:31:09.737329 13281 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3194: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 10:31:09.737515 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3194
2026-09-25T10:31:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1954 ms)
[----------] 18 tests from AuthTokenIssuingTest (36085 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:09 dist-test-slave-f9rl krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:09 dist-test-slave-f9rl krb5kdc[3292](info): set up 2 sockets
Sep 25 10:31:09 dist-test-slave-f9rl krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260925 10:31:10.851792 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:11 dist-test-slave-f9rl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:11.126600 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38275
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:11.466579 3305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:11.466969 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:11.467095 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:11.467233 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:11.474625 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:11.474758 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:11.474884 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:11.474969 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:11.475083 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:11.485191 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:11.486724 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:11.488070 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:11.500581 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:11.500540 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:11.500635 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:11.501592 3305 server_base.cc:1034] running on GCE node
I20260925 10:31:11.502435 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:11.503881 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:11.505105 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790332271505042 us; error 34 us; skew 500 ppm
I20260925 10:31:11.505496 3305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:11 dist-test-slave-f9rl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:11.508747 3305 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:11.510366 3305 webserver.cc:460] Webserver started at http://127.12.248.126:46163/ using document root <none> and password file <none>
I20260925 10:31:11.511086 3305 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:11.511222 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:11.511590 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:11.514720 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "e0f0224b63b149ea9bb91f07e9a3fce1"
format_stamp: "Formatted at 2026-09-25 10:31:11 on dist-test-slave-f9rl"
server_key: "99522f6cf31999d2125d7d7fad5396e9"
server_key_iv: "15c9395689d9dbc6c3376c33c38f041e"
server_key_version: "encryptionkey@0"
I20260925 10:31:11.515542 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "e0f0224b63b149ea9bb91f07e9a3fce1"
format_stamp: "Formatted at 2026-09-25 10:31:11 on dist-test-slave-f9rl"
server_key: "99522f6cf31999d2125d7d7fad5396e9"
server_key_iv: "15c9395689d9dbc6c3376c33c38f041e"
server_key_version: "encryptionkey@0"
I20260925 10:31:11.520905 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 10:31:11.524725 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:11.526230 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:31:11.526458 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "e0f0224b63b149ea9bb91f07e9a3fce1"
format_stamp: "Formatted at 2026-09-25 10:31:11 on dist-test-slave-f9rl"
server_key: "99522f6cf31999d2125d7d7fad5396e9"
server_key_iv: "15c9395689d9dbc6c3376c33c38f041e"
server_key_version: "encryptionkey@0"
I20260925 10:31:11.526696 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:11.538209 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:11.542490 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:11.542773 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:11.562605 3305 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:38275
I20260925 10:31:11.562640 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:38275 every 8 connection(s)
I20260925 10:31:11.564036 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:11.568233 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:11.572808 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3305
I20260925 10:31:11.573040 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:11.573753 13281 external_mini_cluster.cc:1428] Setting key b3780546d933b3f8387757558779bcc3
I20260925 10:31:11.575543 3372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:11 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.125@KRBTEST.COM; defaulting to no policy
I20260925 10:31:11.603581 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.579099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38275 (local address 127.0.0.1:52092)
0925 10:31:11.580468 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.580514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:11.583456 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:11.584415 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:11.584426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:11.585323 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:11.585908 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:11.585937 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:11.587587 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:11.587596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:11.589736 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:11.589755 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:11.590740 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:11.590751 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:11.590986 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:11.591763 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:11.591804 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:11.595178 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:11.598825 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:11.598843 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:11.598858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:11.599515 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:11.599938 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:11.599946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:11.599950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:11.600180 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:11.600968 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:11.600976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:11.601210 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 10:31:11.601541 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 10:31:11.602164 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":137,"spinlock_wait_cycles":46336,"thread_start_us":845,"threads_started":1}
I20260925 10:31:11.603612 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.579099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.12.248.126:38275)
0925 10:31:11.580029 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:11.580046 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:11.580677 (+ 631us) server_negotiation.cc:407] Connection header received
0925 10:31:11.583899 (+ 3222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:11.583905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:11.584145 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:11.584311 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:11.586156 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:11.587405 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:11.589902 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:11.590455 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:11.590586 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:11.595538 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:11.595546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:11.595557 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:11.595587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:11.598667 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:11.599661 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:11.599667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:11.599670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:11.599808 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:11.600304 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:11.600307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:11.600309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:11.600830 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:11.601050 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:11.601675 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 10:31:11.602013 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":396,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.125@KRBTEST.COM" created.
I20260925 10:31:11.607112 3372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:11.608475 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.607960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48380)
0925 10:31:11.608171 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.608306 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 10:31:11.608808 3324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
I20260925 10:31:11.610601 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.610124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48382)
0925 10:31:11.610309 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.610412 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 10:31:11.610837 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:41675: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
I20260925 10:31:11.638265 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:41675
--webserver_interface=127.12.248.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:31:11.660231 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } attempt: 1
I20260925 10:31:11.661702 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.661025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48384)
0925 10:31:11.661288 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.661495 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:31:11.662902 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.662447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48394)
0925 10:31:11.662627 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.662728 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 10:31:11.663120 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:41675: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
I20260925 10:31:11.731448 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } attempt: 2
I20260925 10:31:11.732792 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.732186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48396)
0925 10:31:11.732494 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.732618 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 10:31:11.734128 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.733613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48398)
0925 10:31:11.733815 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.733919 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:31:11.734427 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:41675: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
I20260925 10:31:11.825713 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } attempt: 3
I20260925 10:31:11.827059 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.826537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48404)
0925 10:31:11.826777 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.826890 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 10:31:11.828145 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.827749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48414)
0925 10:31:11.827901 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.827983 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:31:11.828357 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:41675: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
I20260925 10:31:11.971604 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } attempt: 4
I20260925 10:31:11.972942 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.972338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48426)
0925 10:31:11.972655 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.972763 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260925 10:31:11.974088 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:11.973680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48430)
0925 10:31:11.973847 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:11.973942 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:31:11.974327 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:41675: Network error: Client connection negotiation failed: client connection to 127.12.248.125:41675: connect: Connection refused (error 111)
W20260925 10:31:11.977968 3379 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:11.978333 3379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:11.978420 3379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:11.978591 3379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:11.986755 3379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:11.986878 3379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:11.986979 3379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:11.987075 3379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:11.987164 3379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:11.997893 3379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.248.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:11.999361 3379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:12.000766 3379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:12.013551 3384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:12.013590 3387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:12.013515 3385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:12.014721 3379 server_base.cc:1034] running on GCE node
I20260925 10:31:12.015573 3379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:12.017025 3379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:12.018273 3379 hybrid_clock.cc:648] HybridClock initialized: now 1790332272018200 us; error 45 us; skew 500 ppm
I20260925 10:31:12.018635 3379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:12.021982 3379 init.cc:377] Logged in from keytab as kudu/127.12.248.125@KRBTEST.COM (short username kudu)
I20260925 10:31:12.023602 3379 webserver.cc:460] Webserver started at http://127.12.248.125:33371/ using document root <none> and password file <none>
I20260925 10:31:12.024343 3379 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:12.024474 3379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:12.024798 3379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:12.028004 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data/instance:
uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "97a45a59b31550b6c8feae2a716818d5"
server_key_iv: "8d1140ac0bcd9672b805da30d1ecda09"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.028817 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal/instance:
uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "97a45a59b31550b6c8feae2a716818d5"
server_key_iv: "8d1140ac0bcd9672b805da30d1ecda09"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.034344 3379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 10:31:12.037918 3394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:12.039323 3379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 10:31:12.039532 3379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "97a45a59b31550b6c8feae2a716818d5"
server_key_iv: "8d1140ac0bcd9672b805da30d1ecda09"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.039852 3379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:12.053619 3379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:12.058385 3379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:12.058638 3379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:12.078845 3379 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.125:41675
I20260925 10:31:12.078867 3445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.125:41675 every 8 connection(s)
I20260925 10:31:12.080446 3379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
I20260925 10:31:12.084432 3446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:12.084697 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3379
I20260925 10:31:12.084920 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-1/wal/instance
I20260925 10:31:12.085516 13281 external_mini_cluster.cc:1428] Setting key bd8e7073993f7a9ce2d484005b4232ff
I20260925 10:31:12.092015 3446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.124@KRBTEST.COM" created.
I20260925 10:31:12.115572 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.094872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.12.248.126:38275)
0925 10:31:12.095155 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.095162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.096530 (+ 1368us) server_negotiation.cc:407] Connection header received
0925 10:31:12.098788 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.098794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.098897 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.099021 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.100623 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.101410 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.103247 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.104074 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.104252 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.108550 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.108556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.108559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.108597 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.111795 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.112730 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.112737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.112743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.112880 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.113343 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.113349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.113353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.113804 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.114113 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.115161 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.115339 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 10:31:12.117476 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.095543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38275 (local address 127.0.0.1:52106)
0925 10:31:12.096356 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.096402 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.098477 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.099274 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.099283 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.099841 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.100440 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.100457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.101748 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.101752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.103067 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.103080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.104363 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.104370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.104616 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.105419 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.105499 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.108286 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.111960 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.111969 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.111988 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.112591 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.113006 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.113012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.113016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.113217 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.114060 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.114069 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.114587 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.115058 (+ 471us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.115702 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":339,"threads_started":1}
I20260925 10:31:12.119447 3446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:12.134312 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.120340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48440)
0925 10:31:12.120513 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.120532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.120663 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.121496 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.121502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.121956 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.122094 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.122102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.123514 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.123519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.124366 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.124377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.125190 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.125195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.125394 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.126142 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.126158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.128174 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.131701 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.131706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.131713 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.132160 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.132529 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.132535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.132538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.132675 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.133275 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.133282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.133420 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.133616 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.133895 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260925 10:31:12.134367 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.120430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.12.248.125:41675)
0925 10:31:12.121054 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.121060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.121084 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:31:12.121125 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.121130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.121309 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.121489 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.122293 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.123377 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.124504 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.125044 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.125182 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.128389 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.128400 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.128414 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.128441 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.131576 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.132292 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.132303 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.132310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.132423 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.132776 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.132783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.132786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.133151 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.133351 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.133781 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.133995 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":482,"thread_start_us":318,"threads_started":1}
I20260925 10:31:12.137132 3446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:12.138324 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.137886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44312)
0925 10:31:12.138076 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.138173 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 10:31:12.138645 3398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.140331 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.139998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44328)
0925 10:31:12.140159 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.140233 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 10:31:12.140597 3446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
I20260925 10:31:12.152729 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:40753
--webserver_interface=127.12.248.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:31:12.189934 3446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 1
I20260925 10:31:12.191167 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.190702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44340)
0925 10:31:12.190898 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.191018 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 10:31:12.192118 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.191816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44350)
0925 10:31:12.191951 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.192022 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 10:31:12.192350 3446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.260610 3446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 2
I20260925 10:31:12.261921 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.261386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44362)
0925 10:31:12.261668 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.261755 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 10:31:12.263033 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.262703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44368)
0925 10:31:12.262843 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.262914 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 10:31:12.263273 3446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.273567 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } attempt: 5
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:12.287813 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.274321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48446)
0925 10:31:12.274528 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.274555 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.274658 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.275000 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.275003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.275402 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.275604 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.275612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.276981 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.276985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.277959 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.277971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.278899 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.278903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.279058 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.279688 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.279705 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.281802 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.285542 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.285546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.285549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.285933 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.286315 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.286318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.286320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.286459 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.287045 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.287048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.287197 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.287310 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.287541 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:31:12.287987 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.274425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.12.248.125:41675)
0925 10:31:12.274635 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.274640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.274664 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:31:12.274763 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.274768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.274844 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.274952 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.275772 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.276842 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.278100 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.278757 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.278886 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.282036 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.282045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.282052 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.282102 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.285379 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.286052 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.286057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.286061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.286206 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.286565 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.286569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.286573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.286934 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.287081 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.287454 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.287672 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:31:12.289001 3372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:12.290001 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.289650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44378)
0925 10:31:12.289803 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.289881 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:31:12.290877 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.290545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44390)
0925 10:31:12.290709 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.290767 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:12.291101 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.342319 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 1
I20260925 10:31:12.343676 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.343036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44400)
0925 10:31:12.343318 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.343470 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 10:31:12.344918 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.344526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44406)
0925 10:31:12.344681 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.344783 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:31:12.345196 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.354488 3446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 3
I20260925 10:31:12.355650 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.355208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44412)
0925 10:31:12.355402 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.355498 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 10:31:12.356712 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.356291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44420)
0925 10:31:12.356526 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.356599 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260925 10:31:12.356986 3446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
I20260925 10:31:12.413471 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 2
I20260925 10:31:12.414819 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.414193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44432)
0925 10:31:12.414458 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.414584 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 10:31:12.415922 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.415479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44442)
0925 10:31:12.415653 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.415756 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:31:12.416144 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
W20260925 10:31:12.498278 3453 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:12.498589 3453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:12.498651 3453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:12.498737 3453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:31:12.500196 3446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 4
I20260925 10:31:12.501331 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.500844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44448)
0925 10:31:12.501086 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.501180 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:31:12.502382 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.502068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44450)
0925 10:31:12.502193 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.502263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:12.502650 3446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
W20260925 10:31:12.506798 3453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:12.506886 3453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:12.507032 3453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:12.507108 3453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:12.507216 3453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:12.517477 3453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:38275,127.12.248.125:41675,127.12.248.124:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.248.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:12.518831 3453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:12.520213 3453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:31:12.522370 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 3
I20260925 10:31:12.523586 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.523003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44464)
0925 10:31:12.523247 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.523367 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 10:31:12.524664 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.524275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44478)
0925 10:31:12.524439 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.524536 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 10:31:12.524894 3372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:40753: Network error: Client connection negotiation failed: client connection to 127.12.248.124:40753: connect: Connection refused (error 111)
W20260925 10:31:12.533285 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:12.533332 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:12.533244 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:12.534459 3453 server_base.cc:1034] running on GCE node
I20260925 10:31:12.535300 3453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:12.536728 3453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:12.537971 3453 hybrid_clock.cc:648] HybridClock initialized: now 1790332272537917 us; error 27 us; skew 500 ppm
I20260925 10:31:12.538309 3453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:12.541695 3453 init.cc:377] Logged in from keytab as kudu/127.12.248.124@KRBTEST.COM (short username kudu)
I20260925 10:31:12.543453 3453 webserver.cc:460] Webserver started at http://127.12.248.124:42525/ using document root <none> and password file <none>
I20260925 10:31:12.544288 3453 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:12.544432 3453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:12.544836 3453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:12.548228 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data/instance:
uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "9eb205642c3ac5b66bedd06858b97cc8"
server_key_iv: "6d25b45687d4b41fe4da2748f8085e6b"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.549103 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal/instance:
uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "9eb205642c3ac5b66bedd06858b97cc8"
server_key_iv: "6d25b45687d4b41fe4da2748f8085e6b"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.554817 3453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 10:31:12.558720 3468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:12.560074 3453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 10:31:12.560256 3453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19"
format_stamp: "Formatted at 2026-09-25 10:31:12 on dist-test-slave-f9rl"
server_key: "9eb205642c3ac5b66bedd06858b97cc8"
server_key_iv: "6d25b45687d4b41fe4da2748f8085e6b"
server_key_version: "encryptionkey@0"
I20260925 10:31:12.560494 3453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:12.580587 3453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:12.592706 3453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:12.592939 3453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:12.611977 3453 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.124:40753
I20260925 10:31:12.611996 3519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.124:40753 every 8 connection(s)
I20260925 10:31:12.613395 3453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
I20260925 10:31:12.617429 3520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:12.618646 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3453
I20260925 10:31:12.618880 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331906218802-13281-0/minicluster-data/master-2/wal/instance
I20260925 10:31:12.619501 13281 external_mini_cluster.cc:1428] Setting key b4982f4e0610ef9c41c7fa42729356e2
I20260925 10:31:12.627741 3520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:31:12.645270 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.621585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.12.248.124:40753)
0925 10:31:12.622394 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.622409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.622451 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:31:12.624357 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.624379 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.624803 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.625149 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.627086 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.628407 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.630374 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.631604 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.632473 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.636541 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.636549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.636560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.636604 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.640474 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.641286 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.641292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.641299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.641541 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.641968 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.641974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.641979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.642672 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.643082 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.643553 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.644031 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":410,"threads_started":1}
I20260925 10:31:12.648437 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.630697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:38275 (local address 127.0.0.1:52114)
0925 10:31:12.631637 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.631698 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.631900 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.632648 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.632658 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.633477 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.633756 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.633765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.635165 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.635172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.636659 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.636674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.637620 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.637626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.637820 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.638492 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.638518 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.641304 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.644607 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.644619 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.644626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.645136 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.646530 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.646534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.646536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.646667 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.647291 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.647297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.647466 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.647726 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.648032 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":385,"threads_started":1}
I20260925 10:31:12.648500 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.630760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.12.248.126:38275)
0925 10:31:12.632172 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.632206 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.632231 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:12.632296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.632303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.632407 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.632557 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.633927 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.635010 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.636807 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.637375 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.637577 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.641542 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.641547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.641551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.641583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.644482 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.645261 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.645268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.645274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.645441 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.646793 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.646797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.646800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.647137 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.647283 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.647850 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.648076 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1169,"threads_started":1}
I20260925 10:31:12.650125 3520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:12.666940 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.650963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:41675 (local address 127.0.0.1:48458)
0925 10:31:12.651130 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.651151 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.651250 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.651631 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.651635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.652021 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.652173 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.652181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.653700 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.653708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.655624 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.655643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.657234 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.657242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.657577 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.658260 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.658291 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.661175 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.664254 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.664261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.664265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.664629 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.665121 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.665128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.665134 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.665339 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.665930 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.665938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.666105 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.666266 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.666573 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 10:31:12.667042 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.648555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.12.248.126:38275)
0925 10:31:12.648777 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.648782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.649094 (+ 312us) server_negotiation.cc:407] Connection header received
0925 10:31:12.649159 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.649187 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.649295 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.649401 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.652635 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.653787 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.655122 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.655616 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.655761 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.660495 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.660504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.660512 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.660564 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.664065 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.664918 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.664921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.664924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.665099 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.665572 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.665575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.665577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.665842 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.666138 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.666501 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.666716 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":12430,"threads_started":1}
I20260925 10:31:12.667269 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.650984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.12.248.125:41675)
0925 10:31:12.651228 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.651237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.651262 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:12.651374 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.651383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.651482 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.651596 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.652345 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.653512 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.655840 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.656792 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.656914 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.661414 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.661418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.661421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.661529 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.664113 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.664806 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.664812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.664815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.665001 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.665457 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.665463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.665466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.665799 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.665917 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.666419 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.666662 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 10:31:12.668246 3520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:12.682962 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.668941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44484)
0925 10:31:12.669118 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.669138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.669242 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.669749 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.669755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.670263 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.670457 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.670469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.671568 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.671574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.672548 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.672559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.673608 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.673614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.673818 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.674441 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.674456 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.676619 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.680176 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.680182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.680186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.680708 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.681106 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.681111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.681115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.681250 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.681851 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.681856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.682000 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.682171 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.682490 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 10:31:12.683007 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.669313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.12.248.124:40753)
0925 10:31:12.669479 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.669485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.669500 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:31:12.669551 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.669554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.669603 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.669677 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.670630 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.671435 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.672692 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.673410 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.673570 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.676806 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.676815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.676821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.676866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.680025 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.680835 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.680843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.680848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.680980 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.681361 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.681370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.681373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.681714 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.681873 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.682260 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.682655 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":37}
I20260925 10:31:12.684396 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.668808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.12.248.125:41675)
0925 10:31:12.668972 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.668979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.668995 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:31:12.669163 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.669168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.669236 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.669331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.670891 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.671776 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.673307 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.673918 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.674059 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.679144 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.679157 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.679161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.679190 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.681798 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.682433 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.682440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.682448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.682578 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.682967 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.682973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.682976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.683347 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.683638 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.683912 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.684149 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:31:12.689576 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:12.689572 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: Bootstrap starting.
I20260925 10:31:12.692968 3520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:12.694171 3520 log.cc:824] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:12.692116 3347 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52128
I20260925 10:31:12.697290 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: No bootstrap required, opened a new log
I20260925 10:31:12.702114 3372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 4
I20260925 10:31:12.706544 3520 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.706935 3520 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b7e7e734b64e03aa2a16fd95ba4a19, State: Initialized, Role: FOLLOWER
I20260925 10:31:12.707561 3520 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.708705 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.709159 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:12.709966 3520 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332271, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:31:12.720249 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.702911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.12.248.124:40753)
0925 10:31:12.703118 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.703125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.703148 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:31:12.703214 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.703219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.703320 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.703460 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.704489 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.705813 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.707525 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.708368 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.708523 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.712799 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.712804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.712807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.712837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.716765 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.717730 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.717735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.717738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.717906 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.718442 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.718448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.718450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.718837 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.719033 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.719590 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.719836 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:31:12.720474 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.702825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44500)
0925 10:31:12.703017 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.703035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.703129 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.703461 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.703464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.704102 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.704329 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.704340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.705967 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.705972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.707370 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.707383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.708538 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.708549 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.708760 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.709539 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.709561 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.712592 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.716918 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.716928 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.716934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.717572 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.718047 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.718055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.718060 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.718295 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.719012 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.719020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.719260 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.719467 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.719891 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:31:12.725245 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: Bootstrap starting.
I20260925 10:31:12.718367 3453 master_runner.cc:186] Error getting master registration for 127.12.248.126:38275: OK, instance_id { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" instance_seqno: 1790332271556775 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:31:12.727952 3453 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:31:12.729208 3539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:31:12.729331 3539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:31:12.729691 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:12.731118 3372 log.cc:824] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:12.744813 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: No bootstrap required, opened a new log
I20260925 10:31:12.753608 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.754106 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f0224b63b149ea9bb91f07e9a3fce1, State: Initialized, Role: FOLLOWER
I20260925 10:31:12.754693 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.755754 3542 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.756171 3542 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:12.757300 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:12.757378 3347 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:31:12.757639 3347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:12.760200 3347 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.761579 3347 leader_election.cc:288] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [CANDIDATE]: Term 1 election: Requested vote from peers c5e0c8d6ad6d4622a3755b3ee1e0e1e3 (127.12.248.125:41675), a4b7e7e734b64e03aa2a16fd95ba4a19 (127.12.248.124:40753)
I20260925 10:31:12.763288 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19"
I20260925 10:31:12.763926 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:12.768656 3495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f0224b63b149ea9bb91f07e9a3fce1 in term 1.
I20260925 10:31:12.769730 3324 leader_election.cc:302] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b7e7e734b64e03aa2a16fd95ba4a19, e0f0224b63b149ea9bb91f07e9a3fce1; no voters:
I20260925 10:31:12.770371 3542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:12.772189 3542 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [term 1 LEADER]: Becoming Leader. State: Replica: e0f0224b63b149ea9bb91f07e9a3fce1, State: Running, Role: LEADER
I20260925 10:31:12.772883 3542 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.779100 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f0224b63b149ea9bb91f07e9a3fce1. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.779531 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:12.791131 3548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 10:31:12.795758 3326 tablet.cc:2369] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:31:12.802284 3446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } attempt: 5
Sep 25 10:31:12 dist-test-slave-f9rl krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:31:12.823585 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.802989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:40753 (local address 127.0.0.1:44516)
0925 10:31:12.803440 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:12.803463 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:12.803585 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:12.804670 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:12.804677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:12.805329 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:12.805618 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.805635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.807208 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.807218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.809049 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:12.809070 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.810659 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.810668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:12.810941 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.811784 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:12.811815 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:12.815184 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:12.820173 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.820182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.820187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.820855 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.821393 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:12.821401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:12.821407 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:12.821605 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:12.822404 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:12.822412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:12.822596 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 10:31:12.822764 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:31:12.823117 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260925 10:31:12.824712 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.803836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.12.248.124:40753)
0925 10:31:12.804062 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.804071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.804092 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:31:12.804164 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.804172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.804265 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.804397 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.805822 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.807002 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.809259 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.810240 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.810393 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.815484 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.815494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.815501 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.815557 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.819953 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.821017 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.821026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.821032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.821227 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.821753 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.821765 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.821769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.822168 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.822323 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.823981 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.824241 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 10:31:12.831120 3446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: Bootstrap starting.
I20260925 10:31:12.832996 3548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:12.841313 3446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:12.842916 3446 log.cc:824] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: Log is configured to *not* fsync() on all Append() calls
W20260925 10:31:12.845022 3474 tablet.cc:2369] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:31:12.848871 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.790082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.12.248.126:38275)
0925 10:31:12.790482 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.790490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.794329 (+ 3839us) server_negotiation.cc:407] Connection header received
0925 10:31:12.801887 (+ 7558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.801894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.802004 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.802096 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.832885 (+ 30789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.834160 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.836088 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.836839 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.836951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.840539 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.840547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.840553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.840599 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.844657 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.845598 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.845605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.845608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.845773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.846226 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.846232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.846235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.846656 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.846900 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.848135 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.848407 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260925 10:31:12.852121 3446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: No bootstrap required, opened a new log
I20260925 10:31:12.858660 3565 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:31:12.861404 3495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [term 1 FOLLOWER]: Refusing update from remote peer e0f0224b63b149ea9bb91f07e9a3fce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:31:12.865001 3546 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:31:12.877769 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.790072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.12.248.125:41675)
0925 10:31:12.790270 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.790278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.822980 (+ 32702us) server_negotiation.cc:407] Connection header received
0925 10:31:12.824556 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.824562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.824639 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.824711 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.844563 (+ 19852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.845858 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.859101 (+ 13243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.859917 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.860036 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.869090 (+ 9054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.869098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.869106 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.869162 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.873656 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.874792 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.874800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.874806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.874990 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.875558 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.875565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.875568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.876051 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.876314 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.877008 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.877280 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 10:31:12.885156 3446 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.887230 3446 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e0c8d6ad6d4622a3755b3ee1e0e1e3, State: Initialized, Role: FOLLOWER
I20260925 10:31:12.888373 3446 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } }
I20260925 10:31:12.895069 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f0224b63b149ea9bb91f07e9a3fce1. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.895455 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:12.865902 3305 master_runner.cc:186] Error getting master registration for 127.12.248.125:41675: OK, instance_id { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" instance_seqno: 1790332272073159 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:31:12.897166 3305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:31:12.899742 3566 mvcc.cc:205] Tried to move back new op lower bound from 7333200989618847744 to 7333200989285847040. Current Snapshot: MvccSnapshot[applied={T|T < 7333200989618847744}]
I20260925 10:31:12.900643 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:12.806712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.12.248.124:40753)
0925 10:31:12.828354 (+ 21642us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:12.828361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:12.828568 (+ 207us) server_negotiation.cc:407] Connection header received
0925 10:31:12.829191 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:12.829196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:12.829317 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:12.829748 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:12.848763 (+ 19015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.850282 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.852360 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:12.853060 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:12.853192 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:12.858666 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:12.858673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:12.858679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:12.858724 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:12.877987 (+ 19263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.892841 (+ 14854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.892853 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.892856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.893102 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:12.896907 (+ 3805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:12.896913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:12.896917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:12.897344 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:12.897676 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:12.899613 (+ 1937us) server_negotiation.cc:299] Negotiation successful
0925 10:31:12.899900 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20859,"thread_start_us":19876,"threads_started":1}
I20260925 10:31:12.900803 3568 mvcc.cc:205] Tried to move back new op lower bound from 7333200989618847744 to 7333200989285847040. Current Snapshot: MvccSnapshot[applied={T|T < 7333200989618847744}]
I20260925 10:31:12.911438 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.911829 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:12.911744 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.911998 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:12.913368 3548 catalog_manager.cc:1347] Generated new cluster ID: 1c96376b29a7483e96ca50a7b703b93d
I20260925 10:31:12.913592 3548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:12.913892 3570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:12.914351 3570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:12.915863 3446 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:12.916317 3420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:12.915884 3421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3"
I20260925 10:31:12.921384 3420 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [term 1 FOLLOWER]: Refusing update from remote peer e0f0224b63b149ea9bb91f07e9a3fce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:31:12.932857 3421 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0f0224b63b149ea9bb91f07e9a3fce1 in term 1.
I20260925 10:31:12.942226 3542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260925 10:31:12.976413 3548 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:12.978343 3548 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:13.001863 3548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1: Generated new TSK 0
I20260925 10:31:13.003070 3548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:13.005558 3570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f0224b63b149ea9bb91f07e9a3fce1. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:13.006012 3570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:13.007689 3546 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:13.008046 3546 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f0224b63b149ea9bb91f07e9a3fce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:13.022627 3578 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0f0224b63b149ea9bb91f07e9a3fce1" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 38275 } } peers { permanent_uuid: "c5e0c8d6ad6d4622a3755b3ee1e0e1e3" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 41675 } } peers { permanent_uuid: "a4b7e7e734b64e03aa2a16fd95ba4a19" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 40753 } } }
I20260925 10:31:13.024603 3578 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:13.119287 3592 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 10:31:13.140053 3525 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.132610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.12.248.126:38275)
0925 10:31:13.132852 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.132860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.138089 (+ 5229us) server_negotiation.cc:407] Connection header received
0925 10:31:13.138330 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.138340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.138439 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.138518 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.139755 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52156: BlockingRecv error: recv got EOF from 127.0.0.1:52156 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 10:31:13.143142 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.132126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.12.248.125:41675)
0925 10:31:13.132356 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.132366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.141178 (+ 8812us) server_negotiation.cc:407] Connection header received
0925 10:31:13.141464 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.141472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.141567 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.141650 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.142907 (+ 1257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48488: BlockingRecv error: recv got EOF from 127.0.0.1:48488 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:31:13.145007 3560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.142073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.12.248.124:40753)
0925 10:31:13.142278 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.142284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.143151 (+ 867us) server_negotiation.cc:407] Connection header received
0925 10:31:13.143427 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.143433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.143510 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.143579 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.144752 (+ 1173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44544: BlockingRecv error: recv got EOF from 127.0.0.1:44544 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 10:31:13.152377 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.149772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.12.248.125:41675)
0925 10:31:13.150121 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.150129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.150848 (+ 719us) server_negotiation.cc:407] Connection header received
0925 10:31:13.151055 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.151059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.151206 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.151321 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.152102 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48492: BlockingRecv error: recv got EOF from 127.0.0.1:48492 (error 108)
Metrics: {"server-negotiator.queue_time_us":227}
W20260925 10:31:13.160789 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.158568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.12.248.125:41675)
0925 10:31:13.158781 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.158786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.159408 (+ 622us) server_negotiation.cc:407] Connection header received
0925 10:31:13.159598 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.159603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.159688 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.159792 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.160581 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48496: BlockingRecv error: recv got EOF from 127.0.0.1:48496 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260925 10:31:13.171636 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.166819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.12.248.125:41675)
0925 10:31:13.167000 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.167006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.170085 (+ 3079us) server_negotiation.cc:407] Connection header received
0925 10:31:13.170288 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.170297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.170406 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.170521 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.171384 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48506: BlockingRecv error: recv got EOF from 127.0.0.1:48506 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 10:31:13.181248 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.179029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.12.248.125:41675)
0925 10:31:13.179252 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.179260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.179789 (+ 529us) server_negotiation.cc:407] Connection header received
0925 10:31:13.179976 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.179983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.180072 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.180178 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.181001 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48520: BlockingRecv error: recv got EOF from 127.0.0.1:48520 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260925 10:31:13.195111 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.192711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.12.248.125:41675)
0925 10:31:13.192909 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.192916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.193509 (+ 593us) server_negotiation.cc:407] Connection header received
0925 10:31:13.193733 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.193739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.193837 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.193953 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.194872 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48524: BlockingRecv error: recv got EOF from 127.0.0.1:48524 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:31:13.220643 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.218267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.12.248.125:41675)
0925 10:31:13.218479 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.218485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.219060 (+ 575us) server_negotiation.cc:407] Connection header received
0925 10:31:13.219290 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.219296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.219401 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.219523 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.220390 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48536: BlockingRecv error: recv got EOF from 127.0.0.1:48536 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:31:13.261736 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.259321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.12.248.125:41675)
0925 10:31:13.259514 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.259521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.260109 (+ 588us) server_negotiation.cc:407] Connection header received
0925 10:31:13.260330 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.260337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.260447 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.260566 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.261426 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43284: BlockingRecv error: recv got EOF from 127.0.0.1:43284 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 10:31:13.331239 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.329060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.12.248.125:41675)
0925 10:31:13.329289 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.329298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.329725 (+ 427us) server_negotiation.cc:407] Connection header received
0925 10:31:13.330000 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.330005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.330097 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.330208 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.331016 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43286: BlockingRecv error: recv got EOF from 127.0.0.1:43286 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:31:13.466459 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.464528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.12.248.125:41675)
0925 10:31:13.464792 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.464799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.465032 (+ 233us) server_negotiation.cc:407] Connection header received
0925 10:31:13.465195 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.465200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.465272 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.465421 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.466166 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43294: BlockingRecv error: recv got EOF from 127.0.0.1:43294 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 10:31:13.729054 3450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:13.727236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.12.248.125:41675)
0925 10:31:13.727412 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:13.727416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:13.727701 (+ 285us) server_negotiation.cc:407] Connection header received
0925 10:31:13.727894 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:13.727898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:13.727968 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:13.728062 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:13.728792 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43308: BlockingRecv error: recv got EOF from 127.0.0.1:43308 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:31:13.731581 3539 catalog_manager.cc:1259] Loaded cluster ID: 1c96376b29a7483e96ca50a7b703b93d
I20260925 10:31:13.731666 3539 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: loading cluster ID for follower catalog manager: success
I20260925 10:31:13.734273 3539 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: acquiring CA information for follower catalog manager: success
I20260925 10:31:13.736074 3539 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4b7e7e734b64e03aa2a16fd95ba4a19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:31:14.124402 3592 catalog_manager.cc:1259] Loaded cluster ID: 1c96376b29a7483e96ca50a7b703b93d
I20260925 10:31:14.124572 3592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: loading cluster ID for follower catalog manager: success
I20260925 10:31:14.127454 3592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: acquiring CA information for follower catalog manager: success
I20260925 10:31:14.129369 3592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5e0c8d6ad6d4622a3755b3ee1e0e1e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:31:14.253700 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:14.246226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.12.248.125:41675)
0925 10:31:14.246851 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:14.246858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:14.247123 (+ 265us) server_negotiation.cc:407] Connection header received
0925 10:31:14.247337 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:14.247342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:14.247511 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:14.247620 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:14.248137 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.248911 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.250970 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.251540 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.251648 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:14.252249 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:14.252925 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:14.253042 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:14.253201 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:31:14.253277 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":289,"threads_started":1}
I20260925 10:31:14.268527 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:14.261717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.12.248.124:40753)
0925 10:31:14.262546 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:14.262555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:14.262580 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:14.262770 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:14.262775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:14.262939 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:14.263156 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:14.263612 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.264457 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.266047 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.266543 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.266701 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:14.267252 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:14.267841 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:14.268019 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:14.268086 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:31:14.268179 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":565,"threads_started":1}
I20260925 10:31:14.291728 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:14.276723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.12.248.126:38275)
0925 10:31:14.281962 (+ 5239us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:14.281972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:14.282001 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:31:14.282163 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:14.282172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:14.282460 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:14.283596 (+ 1136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:14.283885 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.285119 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.287597 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.288249 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.288720 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:14.289168 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:14.290161 (+ 993us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:14.291143 (+ 982us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:14.291269 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:31:14.291379 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5068,"thread_start_us":4942,"threads_started":1}
I20260925 10:31:14.293819 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:14.277066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.12.248.124:40753)
0925 10:31:14.277304 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:14.277313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:14.285036 (+ 7723us) server_negotiation.cc:407] Connection header received
0925 10:31:14.285390 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:14.285397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:14.285532 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:14.285629 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:14.286620 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.287828 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.290411 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.292187 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.292303 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:14.292950 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:14.293096 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:14.293164 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:14.293409 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:31:14.293515 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 10:31:14.296223 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:14.276256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.12.248.125:41675)
0925 10:31:14.276526 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:14.276533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:14.284024 (+ 7491us) server_negotiation.cc:407] Connection header received
0925 10:31:14.284215 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:14.284221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:14.284325 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:14.284427 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:14.285565 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.286949 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.289499 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:14.291757 (+ 2258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:14.291929 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:14.295298 (+ 3369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:14.295437 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:14.295510 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:14.295786 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 10:31:14.295873 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 10:31:14.297675 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3305
I20260925 10:31:14.299336 3367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:14.457132 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3305
W20260925 10:31:14.477918 3470 connection.cc:535] server connection from 127.0.0.1:44500 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:31:14.478708 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3379
I20260925 10:31:14.479940 3441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:14.596055 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3379
I20260925 10:31:14.616048 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3453
I20260925 10:31:14.617210 3515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:14.727070 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3453
2026-09-25T10:31:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4998 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:31:14 dist-test-slave-f9rl krb5kdc[3674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:31:14 dist-test-slave-f9rl krb5kdc[3674](info): set up 2 sockets
Sep 25 10:31:14 dist-test-slave-f9rl krb5kdc[3674](info): commencing operation
krb5kdc: starting...
W20260925 10:31:15.851627 13281 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/test-user.keytab.
Sep 25 10:31:16 dist-test-slave-f9rl krb5kdc[3674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.248.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
Entry for principal kudu/127.12.248.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab.
I20260925 10:31:16.127437 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44169
--webserver_interface=127.12.248.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:31:16.473526 3687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:16.473912 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:16.474051 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:16.474177 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:31:16.481731 3687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:16.481828 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:16.481922 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:16.482013 3687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:16.482128 3687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:16.492543 3687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.126.keytab
--principal=kudu/127.12.248.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.126:44169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.248.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:16.494009 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:16.495375 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:16.507412 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:16.507366 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:16.507328 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:16.508627 3687 server_base.cc:1034] running on GCE node
I20260925 10:31:16.509562 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:16.510991 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:16.512245 3687 hybrid_clock.cc:648] HybridClock initialized: now 1790332276512187 us; error 36 us; skew 500 ppm
I20260925 10:31:16.512549 3687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:16 dist-test-slave-f9rl krb5kdc[3674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:16.515892 3687 init.cc:377] Logged in from keytab as kudu/127.12.248.126@KRBTEST.COM (short username kudu)
I20260925 10:31:16.517529 3687 webserver.cc:460] Webserver started at http://127.12.248.126:43043/ using document root <none> and password file <none>
I20260925 10:31:16.518261 3687 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:16.518399 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:16.518724 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:16.521795 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data/instance:
uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0"
format_stamp: "Formatted at 2026-09-25 10:31:16 on dist-test-slave-f9rl"
server_key: "e3cde292b2bf36127cdbc6d506146ca8"
server_key_iv: "5b48c4a7c65466b8abad1965f7a871fe"
server_key_version: "encryptionkey@0"
I20260925 10:31:16.522619 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance:
uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0"
format_stamp: "Formatted at 2026-09-25 10:31:16 on dist-test-slave-f9rl"
server_key: "e3cde292b2bf36127cdbc6d506146ca8"
server_key_iv: "5b48c4a7c65466b8abad1965f7a871fe"
server_key_version: "encryptionkey@0"
I20260925 10:31:16.527868 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:31:16.531390 3702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:16.532787 3687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:31:16.532995 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal
uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0"
format_stamp: "Formatted at 2026-09-25 10:31:16 on dist-test-slave-f9rl"
server_key: "e3cde292b2bf36127cdbc6d506146ca8"
server_key_iv: "5b48c4a7c65466b8abad1965f7a871fe"
server_key_version: "encryptionkey@0"
I20260925 10:31:16.533222 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:16.556365 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:16.560626 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:16.560926 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:16.579744 3687 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.126:44169
I20260925 10:31:16.579768 3754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.126:44169 every 8 connection(s)
I20260925 10:31:16.581144 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/data/info.pb
I20260925 10:31:16.583647 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3687
I20260925 10:31:16.583855 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-0/wal/instance
I20260925 10:31:16.584424 13281 external_mini_cluster.cc:1428] Setting key c9e7c8b898951c3856f1ecff2c3e4682
I20260925 10:31:16.585541 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:16.595602 3755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:16 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.125@KRBTEST.COM" created.
I20260925 10:31:16.618362 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.600299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.12.248.126:44169)
0925 10:31:16.601020 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:16.601030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:16.601082 (+ 52us) server_negotiation.cc:407] Connection header received
0925 10:31:16.602284 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:16.602291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:16.602521 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:16.602724 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:16.603987 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:16.605163 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:16.606647 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:16.607202 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:16.607408 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:16.610868 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:16.610876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:16.610888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:16.610916 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:16.613883 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:16.614570 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:16.614582 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:16.614584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:16.614719 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:16.615114 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:16.615126 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:16.615129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:16.615661 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:16.615887 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:16.616410 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 10:31:16.616766 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":514,"thread_start_us":335,"threads_started":1}
I20260925 10:31:16.618394 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.599022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44169 (local address 127.0.0.1:60458)
0925 10:31:16.599773 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.599819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:16.601932 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:16.602743 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:16.602749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:16.603456 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:16.603777 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:16.603795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:16.605353 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:16.605361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:16.606506 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:16.606519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:16.607450 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:16.607456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:16.607718 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:16.608378 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:16.608400 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:16.610637 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:16.614010 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:16.614023 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:16.614037 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:16.614435 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:16.614835 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:16.614842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:16.614847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:16.615004 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:16.615812 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:16.615820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:16.616035 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 10:31:16.616292 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 10:31:16.616897 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":146,"thread_start_us":309,"threads_started":1}
I20260925 10:31:16.622356 3755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:16.623616 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.623119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:38972)
0925 10:31:16.623342 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.623487 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 10:31:16.623904 3703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
I20260925 10:31:16.625576 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.625093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:38980)
0925 10:31:16.625273 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.625365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 10:31:16.625797 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:40219: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
Entry for principal kudu/127.12.248.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab.
I20260925 10:31:16.646562 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:40219
--webserver_interface=127.12.248.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:31:16.675163 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } attempt: 1
I20260925 10:31:16.676441 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.675927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:38984)
0925 10:31:16.676147 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.676286 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 10:31:16.677526 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.677171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:38990)
0925 10:31:16.677301 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.677370 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 10:31:16.677752 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:40219: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
I20260925 10:31:16.746014 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } attempt: 2
I20260925 10:31:16.747282 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.746823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39000)
0925 10:31:16.747052 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.747143 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:31:16.748270 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.747925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39002)
0925 10:31:16.748067 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.748151 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:16.748436 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:40219: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
I20260925 10:31:16.839670 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } attempt: 3
I20260925 10:31:16.840929 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.840488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39018)
0925 10:31:16.840685 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.840779 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:31:16.841966 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.841608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39028)
0925 10:31:16.841754 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.841828 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:31:16.842130 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:40219: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
W20260925 10:31:16.981120 3762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:16.981557 3762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:16.981694 3762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:16.981837 3762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:31:16.985379 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } attempt: 4
I20260925 10:31:16.986711 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.986134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39040)
0925 10:31:16.986480 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.986575 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 10:31:16.987726 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:16.987363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39054)
0925 10:31:16.987512 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:16.987604 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:31:16.987936 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.125:40219: Network error: Client connection negotiation failed: client connection to 127.12.248.125:40219: connect: Connection refused (error 111)
W20260925 10:31:16.990160 3762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:16.990248 3762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:31:16.990396 3762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:16.990475 3762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:16.990582 3762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:31:17.000922 3762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.125.keytab
--principal=kudu/127.12.248.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.125:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.248.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:17.002386 3762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:17.003724 3762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:31:17.015420 3767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:17.015694 3768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:17.015988 3770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:17.016700 3762 server_base.cc:1034] running on GCE node
I20260925 10:31:17.017572 3762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:17.019011 3762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:17.020258 3762 hybrid_clock.cc:648] HybridClock initialized: now 1790332277020179 us; error 54 us; skew 500 ppm
I20260925 10:31:17.020608 3762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:17.023947 3762 init.cc:377] Logged in from keytab as kudu/127.12.248.125@KRBTEST.COM (short username kudu)
I20260925 10:31:17.025497 3762 webserver.cc:460] Webserver started at http://127.12.248.125:43311/ using document root <none> and password file <none>
I20260925 10:31:17.026223 3762 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:17.026361 3762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:17.026708 3762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:17.029831 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data/instance:
uuid: "07d10432620343d1aeda6ef29022e887"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "34c9f829fdd194bcb49944546408f764"
server_key_iv: "36bdee015e47888aa07067b1612162de"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.030663 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal/instance:
uuid: "07d10432620343d1aeda6ef29022e887"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "34c9f829fdd194bcb49944546408f764"
server_key_iv: "36bdee015e47888aa07067b1612162de"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.036322 3762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 10:31:17.040122 3777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:17.041533 3762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:31:17.041721 3762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal
uuid: "07d10432620343d1aeda6ef29022e887"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "34c9f829fdd194bcb49944546408f764"
server_key_iv: "36bdee015e47888aa07067b1612162de"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.041972 3762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:17.059299 3762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:17.063439 3762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:17.063671 3762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:17.083379 3762 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.125:40219
I20260925 10:31:17.083464 3828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.125:40219 every 8 connection(s)
I20260925 10:31:17.084820 3762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/data/info.pb
I20260925 10:31:17.089576 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:17.093297 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3762
I20260925 10:31:17.093539 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-1/wal/instance
I20260925 10:31:17.094086 13281 external_mini_cluster.cc:1428] Setting key 1ee3d203d7fbbe969eb36e7e4e22dd4e
I20260925 10:31:17.098639 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.248.124@KRBTEST.COM; defaulting to no policy
I20260925 10:31:17.122782 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.101385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.12.248.126:44169)
0925 10:31:17.101631 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.101638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.103023 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 10:31:17.106060 (+ 3037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.106064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.106138 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.106234 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.108371 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.109039 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.111492 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.111902 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.111999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.116415 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.116420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.116424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.116460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.119380 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.120298 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.120302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.120304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.120424 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.120906 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.120910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.120912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.121197 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.121364 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.122180 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.122430 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.248.124@KRBTEST.COM" created.
I20260925 10:31:17.124114 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.101985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44169 (local address 127.0.0.1:60464)
0925 10:31:17.102828 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.102893 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.105729 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.106528 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.106542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.107377 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.108179 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.108205 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.109210 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.109218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.111322 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.111340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.112142 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.112151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.112499 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.113298 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.113332 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.116138 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.119647 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.119657 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.119677 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.120146 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.120537 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.120542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.120545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.120770 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.121328 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.121336 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.121795 (+ 459us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.122072 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.122705 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":369,"threads_started":1}
I20260925 10:31:17.126147 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:17.143942 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.127063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39060)
0925 10:31:17.127261 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.127284 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.127413 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.128771 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.128775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.129163 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.129316 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.129326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.131656 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.131660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.132520 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.132532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.134191 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.134198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.134443 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.135199 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.135227 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.137989 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.141307 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.141313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.141319 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.141771 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.142156 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.142160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.142164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.142314 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.142905 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.142909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.143048 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.143202 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.143511 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 10:31:17.144043 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.127378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.12.248.125:40219)
0925 10:31:17.128162 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.128173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.128221 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:31:17.128294 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.128305 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.128590 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.128772 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.129533 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.131497 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.132671 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.134014 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.134136 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.138217 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.138229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.138244 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.138273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.141147 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.141911 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.141922 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.141929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.142039 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.142415 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.142421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.142427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.142780 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.143001 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.143365 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.143669 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":562,"thread_start_us":398,"threads_started":1}
I20260925 10:31:17.147017 3829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:17.148202 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.147699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57246)
0925 10:31:17.147876 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.148034 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:31:17.148454 3778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.150022 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.149694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57252)
0925 10:31:17.149840 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.149925 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:31:17.150213 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
Entry for principal kudu/127.12.248.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab.
I20260925 10:31:17.160429 13281 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
/tmp/dist-test-taskZKvIMz/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:45145
--webserver_interface=127.12.248.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.248.84:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:31:17.199529 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 1
I20260925 10:31:17.200707 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.200259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57268)
0925 10:31:17.200453 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.200559 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 10:31:17.201679 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.201311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57276)
0925 10:31:17.201500 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.201580 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:17.201867 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.270068 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 2
I20260925 10:31:17.271436 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.270926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57288)
0925 10:31:17.271160 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.271254 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 10:31:17.272527 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.272195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57296)
0925 10:31:17.272356 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.272420 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:31:17.272719 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.287201 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } attempt: 5
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:17.302078 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.287978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39062)
0925 10:31:17.288205 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.288228 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.288362 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.288758 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.288762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.289150 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.289340 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.289348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.290764 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.290768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.291635 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.291647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.292659 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.292664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.292851 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.293555 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.293573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.295951 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.299687 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.299690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.299693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.300072 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.300484 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.300487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.300489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.300621 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.301276 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.301279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.301408 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.301582 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.301817 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 10:31:17.302371 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.288182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.12.248.125:40219)
0925 10:31:17.288391 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.288398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.288421 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:31:17.288506 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.288511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.288590 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.288703 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.289515 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.290627 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.291780 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.292503 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.292630 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.296170 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.296176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.296181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.296224 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.299524 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.300191 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.300197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.300200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.300361 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.300738 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.300745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.300748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.301158 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.301312 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.301721 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.301984 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 10:31:17.303414 3755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:17.304404 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.304045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57298)
0925 10:31:17.304207 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.304280 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:31:17.305333 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.305015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57302)
0925 10:31:17.305154 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.305240 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:17.305569 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.356797 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 1
I20260925 10:31:17.358072 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.357581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57314)
0925 10:31:17.357783 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.357906 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:31:17.359050 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.358706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57328)
0925 10:31:17.358865 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.358932 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:31:17.359258 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.363924 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 3
I20260925 10:31:17.365051 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.364607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57330)
0925 10:31:17.364817 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.364908 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 10:31:17.366127 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.365709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57342)
0925 10:31:17.365947 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.366019 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260925 10:31:17.366310 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.427466 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 2
I20260925 10:31:17.428750 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.428243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57358)
0925 10:31:17.428491 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.428582 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 10:31:17.429840 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.429491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57374)
0925 10:31:17.429644 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.429730 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:31:17.430042 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
W20260925 10:31:17.502302 3836 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:31:17.502704 3836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:31:17.502831 3836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:31:17.502964 3836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 10:31:17.509559 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 4
I20260925 10:31:17.510833 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.510302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57380)
0925 10:31:17.510530 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.510621 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260925 10:31:17.511680 3836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:31:17.511798 3836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260925 10:31:17.511893 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.511505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57382)
0925 10:31:17.511650 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.511748 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:31:17.511915 3836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:31:17.512006 3836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:31:17.512094 3836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:31:17.512158 3829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.522312 3836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.248.84:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.248.126:44169,127.12.248.125:40219,127.12.248.124:45145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/krb5kdc/kudu_127.12.248.124.keytab
--principal=kudu/127.12.248.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.248.124:45145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.248.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:40:25 UTC on ccycloud.quasar-gpcgyo.root.comops.site
build id 4672271
ASAN enabled
I20260925 10:31:17.523720 3836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:31:17.525089 3836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:31:17.536239 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 3
I20260925 10:31:17.537736 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.536940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57390)
0925 10:31:17.537237 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.537384 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 10:31:17.538178 3841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:17.538134 3842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:31:17.538951 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.538517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57402)
0925 10:31:17.538710 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.538805 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 10:31:17.538985 3844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:31:17.539170 3755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.248.124:45145: Network error: Client connection negotiation failed: client connection to 127.12.248.124:45145: connect: Connection refused (error 111)
I20260925 10:31:17.540644 3836 server_base.cc:1034] running on GCE node
I20260925 10:31:17.541576 3836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:31:17.543053 3836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:31:17.544312 3836 hybrid_clock.cc:648] HybridClock initialized: now 1790332277544223 us; error 59 us; skew 500 ppm
I20260925 10:31:17.544687 3836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:31:17.548100 3836 init.cc:377] Logged in from keytab as kudu/127.12.248.124@KRBTEST.COM (short username kudu)
I20260925 10:31:17.549633 3836 webserver.cc:460] Webserver started at http://127.12.248.124:42031/ using document root <none> and password file <none>
I20260925 10:31:17.550364 3836 fs_manager.cc:359] Metadata directory not provided
I20260925 10:31:17.550500 3836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:31:17.550843 3836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:31:17.553982 3836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data/instance:
uuid: "a519ea44cd404ac7a1469f90cd475d1d"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "3f1c75cf5caa1a992473a66ef547fdea"
server_key_iv: "5ca4f8279b8be37788add8a3679dfb3c"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.554867 3836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal/instance:
uuid: "a519ea44cd404ac7a1469f90cd475d1d"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "3f1c75cf5caa1a992473a66ef547fdea"
server_key_iv: "5ca4f8279b8be37788add8a3679dfb3c"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.560184 3836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 10:31:17.563782 3851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:31:17.565347 3836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 10:31:17.565590 3836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data,/tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal
uuid: "a519ea44cd404ac7a1469f90cd475d1d"
format_stamp: "Formatted at 2026-09-25 10:31:17 on dist-test-slave-f9rl"
server_key: "3f1c75cf5caa1a992473a66ef547fdea"
server_key_iv: "5ca4f8279b8be37788add8a3679dfb3c"
server_key_version: "encryptionkey@0"
I20260925 10:31:17.565847 3836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:31:17.591084 3836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:31:17.599702 3836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:31:17.599934 3836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:31:17.618191 3836 rpc_server.cc:307] RPC server started. Bound to: 127.12.248.124:45145
I20260925 10:31:17.618209 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.248.124:45145 every 8 connection(s)
I20260925 10:31:17.619514 3836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/data/info.pb
I20260925 10:31:17.623275 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:31:17.626121 13281 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu as pid 3836
I20260925 10:31:17.626307 13281 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZKvIMz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331906218802-13281-0/minicluster-data/master-2/wal/instance
I20260925 10:31:17.626856 13281 external_mini_cluster.cc:1428] Setting key 15365fe5768030b30e598c44df6dd7c0
I20260925 10:31:17.631251 3903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:31:17.650660 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.634228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.126:44169 (local address 127.0.0.1:60478)
0925 10:31:17.635589 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.635633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.635780 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.636138 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.636143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.636816 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.637102 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.637112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.638614 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.638623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.640294 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.640311 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.641088 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.641097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.641270 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.641915 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.641937 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.644036 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.647908 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.647917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.647922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.648352 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.648783 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.648787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.648798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.648966 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.649583 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.649588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.649740 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.649925 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.650141 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":986,"threads_started":1}
I20260925 10:31:17.650674 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.634191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.12.248.126:44169)
0925 10:31:17.635036 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.635045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.635800 (+ 755us) server_negotiation.cc:407] Connection header received
0925 10:31:17.635878 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.635885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.635971 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.636083 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.637251 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.638452 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.640461 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.640850 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.640952 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.644246 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.644255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.644263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.644310 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.647779 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.648476 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.648482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.648486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.648641 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.649095 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.649099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.649102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.649459 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.649602 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.650063 (+ 461us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.650286 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":549,"threads_started":1}
I20260925 10:31:17.650707 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.628779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.12.248.124:45145)
0925 10:31:17.629576 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.629588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.629629 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:31:17.631233 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.631258 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.631642 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.631966 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.633801 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.635593 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.637105 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.638161 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.638329 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.642754 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.642765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.642776 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.642811 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.646433 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.647092 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.647103 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.647112 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.647302 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.647645 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.647650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.647654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.648244 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.648532 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.648936 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.649357 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":356,"threads_started":1}
I20260925 10:31:17.652280 3903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.126@KRBTEST.COM
I20260925 10:31:17.665313 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.653073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.125:40219 (local address 127.0.0.1:39068)
0925 10:31:17.653231 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.653252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.653340 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.653747 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.653751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.654185 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.654375 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.654394 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.655337 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.655345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.656200 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.656211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.656899 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.656904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.657096 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.657680 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.657697 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.659808 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.662931 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.662935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.662938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.663278 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.663679 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.663684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.663688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.663826 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.664333 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.664338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.664495 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.664631 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.664911 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 10:31:17.665347 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.653105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.12.248.125:40219)
0925 10:31:17.653285 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.653294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.653310 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:31:17.653507 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.653514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.653617 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.653717 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.654527 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.655198 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.656349 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.656757 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.656891 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.660000 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.660005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.660009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.660042 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.662795 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.663420 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.663427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.663431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.663566 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.663930 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.663937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.663940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.664208 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.664366 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.664726 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.664974 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 10:31:17.666488 3903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:31:17.668506 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.654274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.12.248.126:44169)
0925 10:31:17.654430 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.654437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.654455 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:31:17.654659 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.654664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.654737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.654835 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.655922 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.656580 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.657888 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.658317 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.658448 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.662713 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.662720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.662726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.662766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.665427 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.666320 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.666324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.666327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.666452 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.666861 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.666865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.666867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.667158 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.667459 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.668019 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.668184 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.124@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.248.125@KRBTEST.COM
I20260925 10:31:17.679904 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.667202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57414)
0925 10:31:17.667376 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.667394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.667489 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.667778 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.667783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.668261 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.668441 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.668454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.669506 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.669512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.670693 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.670709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.671674 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.671678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.671863 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.672417 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.672439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.674666 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.677629 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.677637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.677641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.677976 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.678317 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.678321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.678325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.678448 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.678937 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.678941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.679087 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.679216 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.679512 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:31:17.679955 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.667287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.12.248.124:45145)
0925 10:31:17.667449 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.667453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.667467 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:31:17.667594 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.667598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.667649 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.667721 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.668585 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.669370 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.670825 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.671533 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.671679 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.674847 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.674860 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.674864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.674890 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.677500 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.678089 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.678096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.678101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.678209 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.678545 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.678553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.678556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.678828 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.678974 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.679311 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.679659 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":56}
I20260925 10:31:17.684578 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.670243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.12.248.125:40219)
0925 10:31:17.670414 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.670422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.670441 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:31:17.670580 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.670587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.670660 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.670762 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.672318 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.673421 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.674683 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.675115 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.675251 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.679128 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.679133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.679136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.679165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.682047 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.682676 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.682683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.682687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.682828 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.683195 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.683201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.683204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.683462 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.683750 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.684036 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.684268 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:31:17.684564 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: Bootstrap starting.
I20260925 10:31:17.688385 3903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:17.689199 13281 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:31:17.689615 3903 log.cc:824] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:17.692389 3903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: No bootstrap required, opened a new log
I20260925 10:31:17.691951 3730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60494
I20260925 10:31:17.699359 3903 raft_consensus.cc:348] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.699684 3903 raft_consensus.cc:729] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a519ea44cd404ac7a1469f90cd475d1d, State: Initialized, Role: FOLLOWER
I20260925 10:31:17.700106 3903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.701010 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.701422 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.701989 3903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 10:31:17.712469 3921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:31:17.712610 3921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:31:17.709012 3836 master_runner.cc:186] Error getting master registration for 127.12.248.126:44169: OK, instance_id { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" instance_seqno: 1790332276574460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:31:17.716481 3836 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:31:17.716387 3755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 4
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332276, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.126@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:31:17.731267 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.717047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57426)
0925 10:31:17.717302 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.717318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.717498 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.717849 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.717853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.718270 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.718453 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.718461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.719518 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.719524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.720454 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.720465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.721259 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.721266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.721523 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.722158 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.722175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.724384 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.728410 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.728415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.728420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.728948 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.729344 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.729349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.729352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.729533 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.730239 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.730244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.730425 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.730559 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.730842 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 10:31:17.731354 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.717105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.12.248.124:45145)
0925 10:31:17.717308 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.717315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.717414 (+ 99us) server_negotiation.cc:407] Connection header received
0925 10:31:17.717641 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.717652 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.717709 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.717823 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.718594 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.719377 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.720594 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.721073 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.721206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.724581 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.724588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.724593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.724636 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.728255 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.729079 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.729085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.729089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.729225 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.729645 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.729651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.729656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.730115 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.730292 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.730696 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.730917 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:31:17.737448 3755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: Bootstrap starting.
I20260925 10:31:17.740665 3755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:17.741770 3755 log.cc:824] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:17.744441 3755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: No bootstrap required, opened a new log
I20260925 10:31:17.749645 3755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.750029 3755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a0e7ba6b9e4df8b7929ade5d5e91c0, State: Initialized, Role: FOLLOWER
I20260925 10:31:17.750481 3755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.751423 3924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.751843 3924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.752686 3755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:17.752851 3730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:31:17.753100 3730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:17.756745 3730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.759291 3730 leader_election.cc:288] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [CANDIDATE]: Term 1 election: Requested vote from peers 07d10432620343d1aeda6ef29022e887 (127.12.248.125:40219), a519ea44cd404ac7a1469f90cd475d1d (127.12.248.124:45145)
I20260925 10:31:17.760156 3878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a519ea44cd404ac7a1469f90cd475d1d"
I20260925 10:31:17.760670 3878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:17.764453 3878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a0e7ba6b9e4df8b7929ade5d5e91c0 in term 1.
I20260925 10:31:17.765614 3703 leader_election.cc:302] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a0e7ba6b9e4df8b7929ade5d5e91c0, a519ea44cd404ac7a1469f90cd475d1d; no voters:
I20260925 10:31:17.766148 3924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:31:17.776492 3924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [term 1 LEADER]: Becoming Leader. State: Replica: 39a0e7ba6b9e4df8b7929ade5d5e91c0, State: Running, Role: LEADER
I20260925 10:31:17.777236 3924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.782207 3936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a0e7ba6b9e4df8b7929ade5d5e91c0. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.782691 3936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:17.786643 3939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:31:17.804844 3939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:31:17.811390 3829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } attempt: 5
I20260925 10:31:17.793607 3687 master_runner.cc:186] Error getting master registration for 127.12.248.125:40219: OK, instance_id { permanent_uuid: "07d10432620343d1aeda6ef29022e887" instance_seqno: 1790332277077893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:31:17.814069 3687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 10:31:17.814512 3708 tablet.cc:2369] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:31:17 dist-test-slave-f9rl krb5kdc[3674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790332277, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.248.125@KRBTEST.COM for kudu/127.12.248.124@KRBTEST.COM
I20260925 10:31:17.839054 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.812452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.248.124:45145 (local address 127.0.0.1:57428)
0925 10:31:17.812923 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:31:17.812948 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 10:31:17.813099 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:31:17.815507 (+ 2408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:31:17.815515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:31:17.816211 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:31:17.816507 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.816523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.818644 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.818653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.820551 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:31:17.820570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.822175 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.822184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:31:17.822499 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.823230 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:31:17.823253 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:31:17.826368 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:31:17.832024 (+ 5656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.832032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.832039 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.832748 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.835521 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:31:17.835527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:31:17.835530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:31:17.835711 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:31:17.837872 (+ 2161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:31:17.837877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:31:17.838137 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 10:31:17.838281 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 10:31:17.838619 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":4}
I20260925 10:31:17.843322 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.813405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.12.248.124:45145)
0925 10:31:17.813702 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.813712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.813737 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:31:17.813816 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.813825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.813929 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.814050 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.816864 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.818433 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.820771 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.821742 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.821898 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.826670 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.826679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.826685 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.826739 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.831779 (+ 5040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.834505 (+ 2726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.834514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.834583 (+ 69us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.835257 (+ 674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.836032 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.836037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.836041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.837569 (+ 1528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.839757 (+ 2188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.840174 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.841206 (+ 1032us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 10:31:17.850690 3878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [term 1 FOLLOWER]: Refusing update from remote peer 39a0e7ba6b9e4df8b7929ade5d5e91c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:31:17.850739 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: Bootstrap starting.
W20260925 10:31:17.851927 3857 tablet.cc:2369] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:31:17.851958 3924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:31:17.857672 3829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: Neither blocks nor log segments found. Creating new log.
I20260925 10:31:17.862661 3829 log.cc:824] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: Log is configured to *not* fsync() on all Append() calls
I20260925 10:31:17.878588 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.824809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.12.248.126:44169)
0925 10:31:17.825428 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.825460 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.830146 (+ 4686us) server_negotiation.cc:407] Connection header received
0925 10:31:17.833807 (+ 3661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.833816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.833935 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.834019 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.840789 (+ 6770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.842371 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.850286 (+ 7915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.851020 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.851127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.861885 (+ 10758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.861895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.861902 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.862015 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.866761 (+ 4746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.869551 (+ 2790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.869561 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.869565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.869854 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.873734 (+ 3880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.873740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.873743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.874234 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.874588 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.877606 (+ 3018us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.877989 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260925 10:31:17.880185 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: No bootstrap required, opened a new log
I20260925 10:31:17.891458 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.825180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.12.248.125:40219)
0925 10:31:17.825640 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.825648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.837295 (+ 11647us) server_negotiation.cc:407] Connection header received
0925 10:31:17.837597 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.837605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.837709 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.837812 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.855702 (+ 17890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.856854 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.867733 (+ 10879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.868470 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.868586 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.881673 (+ 13087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.881683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.881691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.881749 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.887152 (+ 5403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.888637 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.888643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.888646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.888853 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.889355 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.889362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.889366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.889850 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.890108 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.890779 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.891034 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":302}
I20260925 10:31:17.900341 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:17.830170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.12.248.124:45145)
0925 10:31:17.853775 (+ 23605us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:17.853781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:17.853802 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:31:17.853909 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:17.853915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:17.854050 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:17.854239 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:17.858876 (+ 4637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.860671 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.866541 (+ 5870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:17.867796 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:17.867988 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:17.888619 (+ 20631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:31:17.888629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:31:17.888634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:31:17.888686 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:31:17.896045 (+ 7359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.896928 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.896934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.896938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.897093 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:31:17.897653 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:31:17.897667 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:31:17.897671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:31:17.898078 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:31:17.899466 (+ 1388us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:17.899605 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:31:17.899885 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13981,"thread_start_us":23257,"threads_started":1}
I20260925 10:31:17.902019 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a0e7ba6b9e4df8b7929ade5d5e91c0. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.902348 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.903993 3829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.904459 3829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d10432620343d1aeda6ef29022e887, State: Initialized, Role: FOLLOWER
I20260925 10:31:17.905054 3829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } }
I20260925 10:31:17.909265 3953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.910110 3804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07d10432620343d1aeda6ef29022e887"
I20260925 10:31:17.910738 3953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.910475 3829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:31:17.910897 3936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.911201 3936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:17.910547 3803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:31:17.912328 3939 catalog_manager.cc:1347] Generated new cluster ID: 09b3f730350844f3a3f21d9d59847a53
I20260925 10:31:17.912422 3939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:31:17.913493 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.913734 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.915445 3803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [term 1 FOLLOWER]: Refusing update from remote peer 39a0e7ba6b9e4df8b7929ade5d5e91c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:31:17.918510 3804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39a0e7ba6b9e4df8b7929ade5d5e91c0 in term 1.
I20260925 10:31:17.920568 3924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:31:17.954315 3939 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:31:17.956771 3939 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:31:17.974674 3939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0: Generated new TSK 0
I20260925 10:31:17.976137 3939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:31:17.983526 3953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a0e7ba6b9e4df8b7929ade5d5e91c0. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.984799 3953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.984704 3963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.985317 3963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:31:17.987023 3936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a0e7ba6b9e4df8b7929ade5d5e91c0" member_type: VOTER last_known_addr { host: "127.12.248.126" port: 44169 } } peers { permanent_uuid: "07d10432620343d1aeda6ef29022e887" member_type: VOTER last_known_addr { host: "127.12.248.125" port: 40219 } } peers { permanent_uuid: "a519ea44cd404ac7a1469f90cd475d1d" member_type: VOTER last_known_addr { host: "127.12.248.124" port: 45145 } } }
I20260925 10:31:17.987336 3936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a0e7ba6b9e4df8b7929ade5d5e91c0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:31:18.007573 3971 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260925 10:31:18.065783 3908 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.051914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.12.248.126:44169)
0925 10:31:18.052314 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.052323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.061905 (+ 9582us) server_negotiation.cc:407] Connection header received
0925 10:31:18.063375 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.063390 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.063511 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.063613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.065310 (+ 1697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60506: BlockingRecv error: recv got EOF from 127.0.0.1:60506 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
W20260925 10:31:18.070945 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.052493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.12.248.125:40219)
0925 10:31:18.052715 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.052724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.067981 (+ 15257us) server_negotiation.cc:407] Connection header received
0925 10:31:18.068877 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.068885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.068979 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.069060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.070703 (+ 1643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39102: BlockingRecv error: recv got EOF from 127.0.0.1:39102 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 10:31:18.071494 3906 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.062188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.12.248.124:45145)
0925 10:31:18.062419 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.062427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.069545 (+ 7118us) server_negotiation.cc:407] Connection header received
0925 10:31:18.069832 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.069842 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.069952 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.070038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.071242 (+ 1204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57446: BlockingRecv error: recv got EOF from 127.0.0.1:57446 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 10:31:18.079430 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.077334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.12.248.125:40219)
0925 10:31:18.077550 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.077559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.078015 (+ 456us) server_negotiation.cc:407] Connection header received
0925 10:31:18.078278 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.078283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.078376 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.078485 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.079185 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39116: BlockingRecv error: recv got EOF from 127.0.0.1:39116 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 10:31:18.127636 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.119642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.12.248.125:40219)
0925 10:31:18.119815 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.119822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.125994 (+ 6172us) server_negotiation.cc:407] Connection header received
0925 10:31:18.126297 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.126302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.126378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.126451 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.127426 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39122: BlockingRecv error: recv got EOF from 127.0.0.1:39122 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 10:31:18.136147 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.133621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.12.248.125:40219)
0925 10:31:18.133908 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.133916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.134513 (+ 597us) server_negotiation.cc:407] Connection header received
0925 10:31:18.134751 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.134764 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.134844 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.134962 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.135879 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39132: BlockingRecv error: recv got EOF from 127.0.0.1:39132 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 10:31:18.146055 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.143583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.12.248.125:40219)
0925 10:31:18.143890 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.143899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.144465 (+ 566us) server_negotiation.cc:407] Connection header received
0925 10:31:18.144675 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.144683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.144809 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.144924 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.145848 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39146: BlockingRecv error: recv got EOF from 127.0.0.1:39146 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260925 10:31:18.166244 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.163724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.12.248.125:40219)
0925 10:31:18.163986 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.163997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.164627 (+ 630us) server_negotiation.cc:407] Connection header received
0925 10:31:18.164887 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.164892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.164979 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.165098 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.165975 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39158: BlockingRecv error: recv got EOF from 127.0.0.1:39158 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 10:31:18.189078 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.186738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.12.248.125:40219)
0925 10:31:18.186987 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.186997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.187528 (+ 531us) server_negotiation.cc:407] Connection header received
0925 10:31:18.187792 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.187796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.187876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.187990 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.188820 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39168: BlockingRecv error: recv got EOF from 127.0.0.1:39168 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 10:31:18.239105 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.230954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.12.248.125:40219)
0925 10:31:18.231294 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.231303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.236666 (+ 5363us) server_negotiation.cc:407] Connection header received
0925 10:31:18.237020 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.237028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.237141 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.237417 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.238816 (+ 1399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39180: BlockingRecv error: recv got EOF from 127.0.0.1:39180 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260925 10:31:18.311358 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.309112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.12.248.125:40219)
0925 10:31:18.309342 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.309351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.309938 (+ 587us) server_negotiation.cc:407] Connection header received
0925 10:31:18.310178 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.310182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.310258 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.310373 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.311146 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39190: BlockingRecv error: recv got EOF from 127.0.0.1:39190 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 10:31:18.448040 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.445559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.12.248.125:40219)
0925 10:31:18.445786 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.445793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.446408 (+ 615us) server_negotiation.cc:407] Connection header received
0925 10:31:18.446623 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.446627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.446752 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.446848 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.447754 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39200: BlockingRecv error: recv got EOF from 127.0.0.1:39200 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 10:31:18.712893 3833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:31:18.711037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.12.248.125:40219)
0925 10:31:18.711263 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:18.711269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:18.711405 (+ 136us) server_negotiation.cc:407] Connection header received
0925 10:31:18.711711 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:18.711716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:18.711791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:18.711871 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:31:18.712655 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39206: BlockingRecv error: recv got EOF from 127.0.0.1:39206 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 10:31:18.714736 3921 catalog_manager.cc:1259] Loaded cluster ID: 09b3f730350844f3a3f21d9d59847a53
I20260925 10:31:18.714823 3921 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: loading cluster ID for follower catalog manager: success
I20260925 10:31:18.717229 3921 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: acquiring CA information for follower catalog manager: success
I20260925 10:31:18.719009 3921 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a519ea44cd404ac7a1469f90cd475d1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:31:19.012102 3971 catalog_manager.cc:1259] Loaded cluster ID: 09b3f730350844f3a3f21d9d59847a53
I20260925 10:31:19.012287 3971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: loading cluster ID for follower catalog manager: success
I20260925 10:31:19.015210 3971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: acquiring CA information for follower catalog manager: success
I20260925 10:31:19.017185 3971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07d10432620343d1aeda6ef29022e887: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:31:19.236361 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:19.230047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.12.248.125:40219)
0925 10:31:19.230636 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:19.230644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:19.230672 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:31:19.230780 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:19.230785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:19.230961 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:19.231101 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:19.231581 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.232448 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.233954 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.234421 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.234572 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:19.235108 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:19.235660 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:19.235847 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:19.235904 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:31:19.235991 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
I20260925 10:31:19.252378 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:19.242589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.12.248.124:45145)
0925 10:31:19.245838 (+ 3249us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:19.245844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:19.245868 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:31:19.246012 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:19.246018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:19.246243 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:19.246469 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:19.247167 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.248313 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.249716 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.250345 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.250485 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:19.251000 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:19.251742 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:19.251896 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:19.251950 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:31:19.252038 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3098,"thread_start_us":2971,"threads_started":1}
I20260925 10:31:19.275774 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:19.260786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.12.248.125:40219)
0925 10:31:19.261031 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:19.261040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:19.268370 (+ 7330us) server_negotiation.cc:407] Connection header received
0925 10:31:19.269141 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:19.269151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:19.269291 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:19.269412 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:19.270016 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.271163 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.273399 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.274114 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.274222 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:19.274911 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:19.275051 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:19.275118 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:19.275353 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 10:31:19.275440 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:31:19.277937 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:19.260679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.12.248.126:44169)
0925 10:31:19.268772 (+ 8093us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:19.268778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:19.270290 (+ 1512us) server_negotiation.cc:407] Connection header received
0925 10:31:19.270605 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:19.270613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:19.270916 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:19.271084 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:19.271940 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.273029 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.275152 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.275753 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.275878 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:19.276617 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:19.277124 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:19.277203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:19.277476 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:31:19.277576 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7945,"thread_start_us":7844,"threads_started":1}
I20260925 10:31:19.283677 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:31:19.269896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.12.248.124:45145)
0925 10:31:19.270068 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:31:19.270076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:31:19.276757 (+ 6681us) server_negotiation.cc:407] Connection header received
0925 10:31:19.277023 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:31:19.277029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:31:19.277129 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:31:19.277191 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:31:19.277799 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.278844 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.281263 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:31:19.282017 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:31:19.282148 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:31:19.282729 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:31:19.282956 (+ 227us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:31:19.283024 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:31:19.283284 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 10:31:19.283386 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 10:31:19.284848 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3687
I20260925 10:31:19.286563 3750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:19.404049 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3687
W20260925 10:31:19.425001 3853 connection.cc:535] server connection from 127.0.0.1:57426 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:31:19.425413 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3762
I20260925 10:31:19.426611 3824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:19.533778 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3762
I20260925 10:31:19.555225 13281 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu pid 3836
I20260925 10:31:19.556459 3898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 10:31:19.667349 13281 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZKvIMz/build/asan/bin/kudu with pid 3836
2026-09-25T10:31:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4939 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9937 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (373394 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST